A quasi-optical mode converter for a whispering-gallery mode gyrotron，consisting of a Vlasov helically-cut launcher and two curved-mirror reflectors, is studied and designed. Firstly, the operation mechanism of the Vlasov type quasi-optical mode converter is investigated using geometric optics theory. And then, applying the aperture field integration and the surface current source integration of the vector diffraction theory, a numerical simulation code is programmed. Finally, the code is used to study a quasi-optical converter system of a W-band whispering-gallery mode gyrotron, and the operation mechanism of the mode conversion in the converter is revealed in detail. Numerical results indicate that the TE12,2 whispering-gallery mode inside the W-band gyrotron is converted into a highly Gaussian-like beam at the output window.