The effective radius of photonic-band-gap cavity (PBGC) is defined, the validity of using it to treat PBGC as a mode selective cylindrical metal cavity is demonstrated, the guiding role of it in the design of PBGC is revealed, and a self-consistent nonlinear theory is established for gyrotron oscillator with PBGC (PBG gyrotron) based on it. The results of theoretical analysis and numerical calculation show that the azimuthal polarized form (traveling wave or standing wave) of RF field has an obvious effect on the beam-wave interacting process and the device operating at second harmonic can achieve higher electron efficiency than that working at fundamental wave under TE-32 mode, which means PBG gyrotron is capable of operating at both high order electromagnetic mode and high order electronic cyclotron harmonic effectively owing to the excellent mode selective ability of PBGC. This gives a new clue to the research of gyrotron oscillator.