The optical nonlinearity, including the nonlinear absorption, the nonlinear refraction and the third-order nonlinear polarizability of 2-(2'-hydroxyphenyl)benzothiazole (HBT) in cyclohexane, has been investigated using a single-beam Z-scan technique. The experiment and calculations show that HBT molecules have obvious two-photon absorption (TPA) effect under the 532 nm picosecond pulsed laser irradiation. The values of the TPA coefficient β0 for different pulse irradiance and the corresponding imaginary part χ(3)I of the third-order nonlinear polarizability were determined according to the open-aperture Z-scan results. Then we measured the nonlinear refractive index γ and the real part χ(3)R of the third-order nonlinear polarizability of HBT, and calculated these two nonlinear parameters with "Gaussian decomposition" method according to the close-aperture Z-scan and open-aperture Z-scan results, when the nonlinear refraction was accompanied by nonlinear absorption. The theoretical results agree very well with the experimental results. This work provides a theoretical as well as a experimental basis for further research and development of the application of the material.