Two-dimensional WTe₂ possesses a special crystal symmetry, leading to novel properties such as quantum spin Hall effect and nonlinear Hall effect. Determining the details of its crystal structure is essential for understanding these interesting properties. Here, we report an optical study on the crystal symmetry of monolayer, bilayer, and trilayer WTe₂, using temperature and polarization dependent Raman spectroscopy and optical second harmonic generation (SHG). We find that monolayer WTe₂ is noncentrosymmetric as indicated by its sizable SHG, in contrast to the commonly believed centrosymmetric 1T' structure. The polarization dependence of the SHG is consistent with the C_s point group. Bilayer WTe₂ exhibits SHG signal more than one order of magnitude higher than in the monolayer and trilayer samples, with its temperature dependence reflecting the ferroelectric phase transition, evidencing strong inversion symmetry breaking induced by layer stacking and interlayer-sliding ferroelectricity. We also observe prominent second-order resonant Raman scattering peaks only in monolayer and bilayer WTe₂, but not in thicker samples, and their temperature dependence indicates an electronic structure highly sensitive to interlayer coupling. These results will be useful for further exploring the properties of atomically thin WTe₂.