Graphene, atomically thin two-dimensional (2D) nanomaterial consisting of a single layer of carbon atoms, has received tremendous attention in the past few decades. Graphene may be considered as an excellent nanomaterial for fabricating nanomechanical resonator systems to investigate the quantum behavior of the motion of micromechanical resonators because of its unique properties of low mass density, high frequency, high quality-factor, and intrinsically small size. Additionally, graphene optomechanics based on a bilayer graphene resonator coupled to a microwave on-chip cavity, where light and micromechanical motion interact via the radiation pressure, has been demonstrtated experimentally recently. In this work, we demonstrate theoretically the nonlinear optical effect including optical bistability and four-wave mixing under the regimes woth different parameters and detunings in a graphene resonator-microwave cavity system. When the graphene optomechanics is driven by one strong pump laser beam, we find that the optical bistability can be controlled by tuning the power and the frequency of the pump beam. The four-wave mixing (FWM) phenomenon is also investigated and we find that sharp peaks in the FWM spectrum exactly are located at the resonant frequency of graphene resonator. Therefore, a straight nonlinear optical means for determining the resonant frequency of the graphene resonator is presented. Setting the cavity field resonating with pump field, and then scanning the probe frequency across the cavity frequency, one can easily and exactly obtain the resonant frequency of the resonator from the FWM spectrum. We further theoretically propose a mass sensor based on the graphene optomechanical system. The mass of external nanoparticles deposited onto the graphene resonator can be measured conveniently by tracking the shift of resonant frequency due to mass changing in the FWM spectrum. Compared with optomechanical mass sensors in linear regime, the nonlinear optical mass sensor may be immune to the detection noise. The system may have potential applications in communication networks for frequency conversion and provide a new platform for high sensitive sensing devices.