Memristors exhibit controllable nonlinear characteristics, generating chaotic signals that are characterized by randomness, sensitivity, and unpredictability, thereby demonstrating significant potential applications in information encryption and signal processing. With the integration of chaos theory and electronic technology, constructing memristive hyperchaotic systems has become a hot topic in nonlinear science and information security. To overcome the limitation of monotonic dynamic characteristics in traditional chaotic systems, we design a novel memristor-based hyperchaotic system with richer dynamic behavior and higher application value in this paper. Moreover, the characteristic analysis, theoretical verification, application exploration, and hardware implementation are conducted to support the engineering applications of the system. Building upon the classical Chen system, this work is innovatively combined with a cubic nonlinear magnetically controlled memristor model as a feedback element. By establishing a mathematical model of the memristor and coupling it with the state equations of the Chen system, we design a four-dimensional memristor-based hyperchaotic system. First, by integrating numerical computation with differential equation theory, a comprehensive mathematical model is established to analyze fundamental properties, such as symmetry and dissipativity, thereby validating the system’s rationality. Second, the system’s dynamical behaviors are analyzed, including attractor phase diagrams, Lyapunov exponents, power spectra, parameter effects, transient dynamics, and coexisting attractors. Simultaneously, variational methods are utilized to analyze unstable periodic orbits within the system. A symbolic coding approach based on orbital characteristics is established to convert orbital information into symbolic sequences, and orbital pruning rules are explored to provide a basis for optimal orbital control. Furthermore, a digital image encryption method is proposed based on this system. Using chaotic sequences as keys, image pixels are scrambled and diffused. The effectiveness of encryption is validated through histogram analysis, correlation analysis, information entropy evaluation, and testing of anti-attack capabilities. Finally, a DSP-based digital circuit hardware platform is constructed to run the system, and the hardware experimental results are compared with software simulation outcomes. These findings reveal that the introduction of memristors induces linearly distributed equilibrium points in phase space, generating hidden attractors that enrich the chaotic behavior of the system. The simulation of dynamic behavior confirms the rich dynamics of this four-dimensional memristor-based hyperchaotic system. The proposed digital image encryption method demonstrates robust security performance. The DSP hardware experiments and software simulations yield highly consistent attractor phase diagrams, validating the correctness and feasibility of the system.