With the rapid development of computer network technology, information security has drawn considerable attention in recent years. Owing to the characteristics of multi-dimensional operation and parallel processing capability, optical image encryption techniques are arousing great interest in many exciting fields. Since the pioneering work on optical image encryption using double random phase encoding technique, a large number of algorithms and architectures have been proposed and realized. However, with the further analysis of the securities of these schemes, most of them have been verified to be vulnerable to different types of attack algorithms. Recently, optical encryption schemes based on the polarization properties of light wave have been extensively studied, for an additional flexibility in the encryption key design is provided, which can achieve high robustness against brute force attack by a combination of multiple private keys. Nevertheless, optical encryption schemes based on the polarization properties of light wave could still be vulnerable to known- and chosen- plaintext attacks. Therefore, in this paper, a novel asymmetric polarization encryption method is implemented for dual images, and combined with interference-based optical image encryption method and a Q-plate. First, the information about the two images to be encrypted is separated into two pure phase plates by means of interference optical image encryption, which will be further encoded into two mutually orthogonally polarized light beams. Next, the Q-plate and pixelated polarizer are used for realizing different polarization distributions of the two light beam. Ultimately, the output intensity distribution is recorded by a charge coupled device (CCD) which will be treated as the final ciphertext. For actualizing the asymmetric encryption, one of the pure phase plates acts as a decryption key, which is different from the encryption key. We can control the polarization state of each pixel according to the parameter q, causing the Q-plate to be electrically controllable and the optic-axis orientation of each pixel to differ from one another. It should be emphasized that the value of q and the polarization angle of the pixelated polarizer play the role of two encryption keys, which improves the security of the algorithm extremely, due to their high sensitiveness. Theoretical analyses and numerical simulations verify the feasibility and effectiveness of the proposed encryption scheme.