Due to their unique nonlinear characteristics and memory effects, memristor-based chaotic systems have become a significant focus of research. However, studies on unstable periodic orbits in memristive chaotic systems are still relatively scarce. In this work, a novel four-dimensional memristive chaotic system is constructed by introducing a trigonometric-function-based memristor to enhance a three-dimensional chaotic system. The dynamical behaviors of the system are analyzed using Lyapunov exponents, Poincaré sections, phase portraits, and time-domain plots. The proposed memristive chaotic system exhibits rich dynamical characteristics, including transient behavior, intermittent chaos, and diverse attractor dynamics under parameter variations. To overcome the limitations of the variational method in finding reliable initial guesses for unstable periodic orbits, an innovative optimization strategy that utilizes the physical properties of trigonometric functions is proposed. Integrated with symbolic dynamics, this strategy can quickly obtain robust initial guesses for unstable periodic orbits within specific intervals. Furthermore, it enables these guesses to migrate into other regions of the attractor, ultimately achieving full coverage of the attractor's unstable periodic orbits. After a systematic analysis of the unstable periodic orbits in the new system, the adaptive backstepping method is employed to control the stability of the known unstable periodic orbits, namely 320 and 0^*1^*3. The pseudorandom sequences from the novel memristive chaotic system pass the U.S. National Institute of Standards and Technology statistical test suite, with all P-values exceeding the 0.01 threshold, which verifies their strong pseudo-random characteristics. Using this system for image encryption results in a key space of 10¹²⁰, significantly enhancing the key space and key sensitivity of the algorithm. The encryption process begins with cross-plane scrambling operations among the red, green and blue channels for initial pixel processing, followed by intra-plane scrambling to further disrupt the pixel arrangement. XOR operations are then employed for pixel value diffusion. The algorithm exhibits outstanding resistance to differential attacks, with average number of pixels change rate and unified average changing intensity values reaching 99.6041% and 33.4933%, respectively. Comprehensive security analyses, including histogram analysis, correlation analysis, resistance to cropping attacks, and runtime evaluation, verify that the proposed encryption scheme not only possesses strong security capabilities but also maintains high computational efficiency, making it highly suitable for practical image encryption applications. Finally, the realizability of the system is verified by utilizing a digital signal processor circuit.