Virtual cathode oscillator, as a kind of space-charge high-power microwave source, has an output microwave mode that is generally an admixture of TM₀₁ mode and TE₁₁ mode. The analysis of the resonator in the anode mesh shows that when the transmission of the anode mesh is high, it is easy to produce strong reflected electron beam, forming a conical quasi-resonator structure, thus enhances the output of TE₁₁ mode. When the transmission of the anode mesh is low, the beam intensity of the reflected electron beam can be weakened due to the absorption of the metal mesh, and the TE₁₁ mode can be suppressed, so the output mode is mainly TM₀₁ mode. In this paper, a feedback coaxial virtual cathode oscillator is investigated with the use of numerical simulation and experimental data analysis. The feedback coaxial virtual cathode oscillator is formed by closing the end of the anode mesh through a metal plate and changing the path of the reflected electron beam from the metal mesh to the gap between cathode and anode. The particle in cell method is used in the numerical simulation of the virtual cathode oscillator, and the impedance of the 400 kV diode is about 13 Ω under a voltage of 400 kV. After the optimal design by numerical simulation, the average output microwave power from the virtual cathode oscillator is 1.5 GW, and the frequency of the microwave is about 4.2 GHz, which is basically consistent with the theoretical calculation results. In this new kind of virtual cathode oscillator, the distribution of reflected electrons is modified by the feedback sheet on the anode mesh, the output high power microwave pattern is demonstrated to be dominated by TM₀₁ mode. The microwave power obtained in the experiment is measured by the array antenna power density integration method. For axisymmetric mode, a receiving antenna array is formed by placing multiple receiving antennas on one side of the axis of the antenna pattern. The power densities of different angles on the horizontal circumference with the phase center of the transmitting antenna are measured, the average power density of two adjacent points is multiplied by the area of the spherical belt between these two points, and then the resulting power is added by the power between the adjacent two points, thereby obtaining the total radiation power. With this method, the microwave power is 850 MW with frequency 4.1 GHz and pulse width 30 ns under slaving voltage 400 kV.