The lattice dynamics of hexagonal perovskite BaVS3 compound is studied by first-principles calculations based on the ab initio pseudopotential and plane wave basis. Phonon spectra throughout the Brillouin zone are obtained using the density-functional perturbation method within the framework of linear response theory. The calculated frequencies of the Raman active modes are compared with the experimental measurements. For E22g and A1g modes, our calculated values are in good agreement with the experiment. But for E32g mode, the value of the calculated frequency differs significantly from the experiment. After further calculation using the frozen-in phonon method, we suggest that this difference is caused by the large anharmonicity of E32g mode. The most prominent feature of the phonon spectrum is that there appears the imaginary frequency for E12u mode, which involves the planar vibration of the vanadium atoms. The appearance of imaginary frequency for E12u mode indicates the planar instability of hexagonal BaVS3 crystal structure. We believe that this soft mode is responsible for the observed hexagonal-orthorhombic phase transition.