FAA Pauses Juniper Router Rollout After Ground Stop, 12,000 Cancellations

The Federal Aviation Administration has paused new Juniper router installations after a telecommunications failure prompted a nearly five-hour nationwide ground stop on November 13. Approximately 12,462 flights were canceled Friday, although the agency says not every cancellation was directly caused by the ground stop, and thousands more flights were delayed as disruption continued through the weekend.

https://youtu.be/zcXP9iTIb1k

The immediate trigger for the operational shutdown was an extraordinary surge of network traffic. According to the FAA’s account of the disruption, its aging telecommunications system became unstable when data flooded the network at a rate of roughly 1,000 connection attempts every four to five seconds. Communications and radar services were disrupted at 17 of the country’s 22 regional air traffic control centers, with instability also spreading into several terminal-control facilities.

The system was not completely offline. Some radio frequencies and radar feeds remained reliable, allowing controllers to communicate with and track aircraft already in the air. Approximately 3,500 aircraft landed safely, while critical medical, law-enforcement and military flights continued operating. The nationwide ground stop prevented additional traffic from entering a network whose available communications and surveillance capacity had become uncertain.

The associated router is not yet a confirmed root cause

The investigation has narrowed the excessive data to two potential originating locations: a radar site in Amarillo, Texas, and a radio communications site in El Dorado, Arkansas. FAA network service provider L3Harris had been conducting routine work at the Arkansas site, including deployment of a new router as part of a transition from obsolete copper connections to fiber optics.

Blocking traffic from the two locations stabilized the network, but identifying the points associated with the flood does not explain why it occurred. The FAA and L3Harris are still investigating the root cause, including the infrastructure’s configuration and performance. Hewlett Packard Enterprise, which owns Juniper, says its equipment was not faulty and operated as configured by the installer.

That leaves several distinct technical questions unresolved: what generated the repeated connection attempts, why they propagated far enough to destabilize services across most regional centers, and how the legacy network responded to the traffic. The FAA says the instability occurred in its legacy time-division-multiplexing system during routine replacement work, rather than in upgraded towers associated with its broader modernization program. It has also found no indication of a cyberattack or outside malicious actor.

A precaution now extends across the router fleet

The FAA says it addressed the initial problem by disabling functionality on the originating Juniper routers. It then disabled the same functionality on approximately 60 other Juniper routers and stopped further installations as a precaution. The agency is also reevaluating when replacement and maintenance work should occur. Work at major control facilities is generally scheduled at night, but the affected installation was at a remote site where similar daytime replacements had previously occurred without systemwide disruption.

The pause reaches beyond a single device because the planned conversion is large. L3Harris is slated to install 3,648 Juniper routers, according to figures cited by senators overseeing aviation policy. Of 400 routers already installed, only 69 were operational. That gap means the FAA still has substantial deployment work ahead, but it also has an opportunity to examine configuration control, testing boundaries and network behavior before placing more devices into service.

Modernizing the network remains necessary because old copper circuits are becoming obsolete and are being disconnected. The operational challenge is to replace them without allowing a local installation or configuration condition to propagate into a national service interruption. Comparable modernization programs can reduce maintenance downtime by supporting controlled transfers to synchronized backup systems, but that benefit depends on testing the combined behavior of new equipment and the legacy architecture around it not merely validating each component separately.

The FAA restored air traffic at 8:15 p.m. after waiting to verify that the network was stable. Its next task is more demanding than locating the routers from which the traffic emerged: investigators must establish why the connection surge spread across 17 control centers and determine what deployment controls are required before a 3,648-router conversion can safely resume.

More aerospace and engineering stories, right in your MSN feed.
Follow AMI on MSN

By Thomas Caldwell — AMI’s senior editor for mechanical and mobility engineering, covering vehicle electronics, systems integration, electrification, chassis systems, propulsion, and safety policy.

Leave a Reply