Reference-frame-independent measurement-device-independent quantum key distribution is adopted to avoid aligning the reference frames in realistic setup, which can guarantee the system security against the slow drift of reference frame. However, the relative motion of reference frame including deviation and fluctuation can influence the performance of reference-frame-independent measurement-device-independent quantum key distribution in practical experimental demonstration. In this paper, taking finite effect into consideration, the performance of reference-frame-independent measurement-device-independent quantum key distribution with biased bases under reference frame deviation and fluctuation is presented to evaluate the effect of the relative motion of reference frame on our scheme, which makes the analysis conform to reality. Our simulation results imply that the key rates fluctuate periodically with the reference frame rotating, while declining with the reference frame fluctuation increasing.