Hierarchical Reinforcement Learning Boosts Air Defense Efficiency

Modern air defense confrontations demand rapid, precise task assignments in environments where threats evolve within seconds. Traditional centralized methods can achieve globally optimal results but often lack the speed to react to sudden changes, while fully distributed approaches may respond quickly but sacrifice coordination. To address this trade-off, researchers have developed a hierarchical reinforcement learning architecture for ground-to-air confrontation (HRL-GC) paired with a model predictive control–proximal policy optimization (MPC-PPO) algorithm.

The HRL-GC framework builds on the one-general-agent-with-multiple-narrow-agents (OGMN) concept, but replaces rule-driven narrow agents with data-driven execution agents. It layers the system into high-level scheduling agents and lower-level execution agents. Scheduling agents coordinate the global situation, assigning targets to execution agents, which then decide the timing, resources, and methods for interception based on local states. This decomposition reduces the complexity of high-dimensional state-action spaces while preserving global coordination.

In the studied red-blue confrontation scenario, the red side defends key assets with long- and short-range interception units, each comprising sensors and interceptors. The blue side launches mixed attacks using cruise missiles, UAVs, fighters, and jammers. Both scheduling and execution agents are modeled as Markov Decision Processes, with state spaces encompassing defender and attacker statuses, action spaces defining tracking and interception choices, and reward functions designed to encourage efficient resource use while maximizing operational effectiveness.

Training efficiency is critical in such complex simulations. Model-free reinforcement learning can be computationally lighter per iteration but suffers from inefficient exploration, especially in the early stages. Model-based approaches, by contrast, learn an environmental model from limited samples, then generate simulated data to accelerate policy learning. The MPC-PPO algorithm leverages this by first training a predictive model using MPC to produce high-quality demonstration datasets. These datasets pre-train the PPO network, reducing the need for costly real-environment interactions early on.

MPC operates by predicting optimal action sequences over a defined horizon, executing the first action, then re-optimizing at each step. This method, combined with PPO’s policy gradient updates, allows agents to blend demonstration data with exploration data in a controlled ratio, gradually shifting toward pure exploration as training progresses to mitigate cumulative model errors.

Safety constraints are integral to the system. Cooperative guidance requirements ensure minimum accuracy and maximum distance thresholds are met during multi-platform engagements. Time-sensitive constraints govern interception timing and sensor activation, factoring in target speed, altitude, and approach geometry. Assignments are only valid when all safety conditions are satisfied.

Experiments were conducted in a digital battlefield simulation environment with randomized physical constraints such as earth curvature effects. Hardware included an Intel Xeon E5-2678v3 CPU and dual Nvidia GeForce 2080Ti GPUs. Execution agents were initially trained using rule-based assignments before transitioning to data-driven learning. Comparative tests with Alpha C2 and OGMN architectures using PPO, A3C, and DDPG algorithms showed HRL-GC achieving higher mean rewards and win ratios more quickly, with more stable curves.

Further comparisons of MPC-PPO against PPO-TAGNA and standard PPO demonstrated that MPC-PPO delivered higher initial rewards and faster win ratio gains in the first 50,000 training steps. Behavioral analyses revealed that untrained agents wasted resources, PPO agents focused narrowly on high-value targets, PPO-TAGNA agents coordinated but with limited scope, while MPC-PPO agents effectively balanced interception of high-threat and high-value targets.

By combining hierarchical reinforcement learning with model-based acceleration, the HRL-GC and MPC-PPO approach enhances both the effectiveness and dynamism of air defense task assignments. The layered agent design preserves global coordination while empowering local autonomy, and the model-based pre-training strategy reduces inefficient exploration, enabling faster convergence toward practical, high-performance policies in large-scale, complex scenarios.