The propagation speed is one of the important parameters of the internal solitary wave (ISW). How to obtain the ISW speed through optical remote sensing images accurately and quickly is an important problem to be solved. In this paper, we simulate ISW optical remote sensing imaging, obtain an experimental database, and build the ISW speed inversion model based on a single-scene optical remote sensing image by using the least squares method and the support vector machine. The accuracy of the ISW speed inversion model is tested by using MODIS Image and GF-4 image data of the South China Sea. The study results are shown below. The least squares ISW speed inversion model can give the regression equation, which is more intuitive and has less accuracy in the water depth ranging from 300 m to 399 m, while the support vector machine ISW speed inversion model has high accuracy in the water depth ranging from 400 m to 1200 m and from 83 m to 299 m. Therefore, the two kinds of ISW speed inversion models have different advantages, and can be applied to the inversion of the ISW speed in the real ocean.