The generation, propagation and application of optical vortex have been hot research topics in recent years. Optical vortex carries orbital angular momentum (OAM) that potentially increases the capacity and the spectral efficiency of optical communication system as a new degree of freedom. The optical vortex can be used not only as information carrier for space-division multiplexing, but also for encoding/decoding. We present a novel free-space optical communication system based on hybrid optical mode array encoding/decoding. The array includes four modes that can easily be identified by image processing. The four modes are Gaussian beam, single optical vortex, and two different composite optical vortices. In this paper, the computer generated hologram (CGH) of the hybrid optical mode array is generated based on the object-oriented conjugate-symmetric extension Fourier holography. When the CGH is loaded onto the electronic addressing reflection-type spatial light modulator (SLM), a single light beam illuminates the SLM, and the desired hybrid optical mode array is generated. In the experiment, a m 32 pixel32 pixel Lena gray image is transferred. At the transmitter, the Lena gray image is scanned line by line. The gray value (0-255) of each pixel with 8-bit information is extracted from the image and converted into a 22 hybrid optical mode array, which is encoded into the CGH. Hence, the m 32 pixel32 pixel Lena gray image is corresponding to a sequence with 1024 CGHs. By switching the CGHs loaded onto the SLM, the Lena gray image is transmitted in the form of the hybrid optical mode array. At the receiver, each hybrid optical mode array is decoded to a pixel value. To distinguish different modes conveniently, two cross lines are set at the center of each mode. By counting the peaks of two intensity distribution lines, the modes can easily be identified. We demonstrate the image reproduction of Lena with zero bit error rate (BER). The experimental result shows the favorable performance of the free-space optical communication link based on hybrid optical mode array encoding/decoding. Compared to that of the traditional single-vortex encoding communication system, the information capacity of our system with 22 hybrid optical mode array increases by four times. In addition, the presented experimental system is feasible and has strong expansibility. The information capacity can increase by 16 times with a 44 hybrid optical mode array based on the same experimental setup. Therefore, the presented free-space optical communication system using hybrid optical mode array encoding/decoding has great significance for improving the capacity of free-space optical communication system.