As most followers of the FAA’s NextGen program know, the backbone of the whole thing will be the en route advanced modernization (Eram) system. Eram is a big project with a $2.15 billion price tag, and it is planned to replace today’s Host upper-airspace computer network, which has supported the NAS for close to 40 years.
Yet amid all the FAA exultation these days about NextGen at ATC conferences and in news releases and public statements by agency officials, Eram scarcely gets a mention. If the FAA were a sitcom, Eram would play the slightly crazy Uncle Edgar who gets hustled off to his room just before guests arrive, to be let out only after they’ve left.
That’s because Eram has become a serious embarrassment for the agency and, presumably, for Lockheed Martin, its builder. Contracted sole source in 2003 and touted as making up for all the shortfalls and problems of the venerable Host, Eram got off to a flying start in 2007, passing the FAA’s acceptance tests at the agency’s Technical Center with a score of more than 95 percent, a major NextGen success. It was Go for the Moon, and Eram installations were forecast to be at all 20 ARTCCs by late 2010, safely ahead of the 2012 end of IBM’s maintenance contract for Host, spare parts for which–along with elderly software experts versed in ancient Host computer lingo like Jovial and Assembler–were becoming extremely scarce. At such a heady time, no one was bothered that FAA acceptance also included responsibility for any additional future costs.
Fast forward to today, and the FAA has little to say about Eram. Uncle Edgar is rarely seen outside his room. In October, top officials from the Government Accountability Office (GAO) and the DOT Inspector General’s Office (IG) testified separately on NextGen before the House subcommittee on aviation. Each testified that Eram’s promised 2010 delivery and its $2.15 billion cost are now history, with delivery now set for 2014 and an extra $350 million added to the cost, with the IG suggesting that 2016 and an additional half a billion might be nearer the mark.
Plagued by System Anomalies
Shortly after Eram made its debut at the Tech Center in 2007, more exacting pre-operational testing at the lower-activity Salt Lake and Seattle ARTCCs started to reveal system anomalies such as misidentifying aircraft, preventing controller “handovers” and a variety of other problems requiring numerous “workarounds” to complete test routines. Primarily, these were identified to originate in the system’s 1.4 million lines of programming software, which were reportedly compounded by the lack of close management of the project’s progress since FAA acceptance. In the IG’s words, “Eram has experienced an almost continuous series of problems since its initial installations at two FAA Control Centers.” The IG also expressed concern about how well Eram would cope with the much heavier demands around high-density centers such as New York and Chicago.
One particularly embarrassing incident occurred in October 2009, when FAA officials were invited to witness the first full switchover from the Host to Eram. After 9.5 hours, escalating Eram problems forced a switch back to the Host. Nevertheless, the IG’s 2011 review revealed that Lockheed Martin had received more than $150 million in performance incentives up to that point “even though Eram was at least $350 million over budget.” In separate testimony, the GAO stated “the FAA’s contract-management vehicle not only supports but rewards and [provides incentives for] poor program-management practices.”
Unfortunately, since Eram is essentially the lynchpin of NextGen, its delays can be expected to create significant cost growth and delays in dependent programs such as ADS-B, DataComm and the system-wide information management (Swim), whose Eram interfaces are currently budgeted at $600 million, plus additional costs and delays in future applications such as trajectory-based operations.
David Grizzle, who earlier replaced Hank Krakowski as head of the FAA’s Air Traffic Organization, recently described Eram as “the chassis on which all the NextGen functionality will be bolted.” It turns out that until all those bolts are securely tightened, the FAA will also be required to pay the collateral damage of keeping the old Host running, at between $7 and $10 million per month, according to the GAO.
Certainly, faults are to be expected during the early introductory period of a system as complex as Eram, but the FAA did not anticipate the number of faults the system has encountered. Both the IG and the GAO cited lax management of the program as a cause of the system’s faults. It is to be hoped that the recently announced amalgamation of all NextGen program managers into a single body, reporting through its yet-to-be-named chief direct to the deputy administrator, can bring order out of the current Eram chaos.
Comments
A controller (not verified)
December 5, 2011 - 2:35am
The FAA has successfully kept a pretty big secret- ERAM failed completely at Salt Lake just a few days before Thanksgiving.
ERAM uses two separate "processes" on a CPU, and two CPUs- so in theory there are four separate programs running at once to carry out a given function.
The failure, though, was typical- since all four processes are identical programs, running on identical inputs (radar, flight plans, etc) it's pretty reasonable that when something makes one crash, all four are going to crash. Sure enough, boom. Total, catastrophic failure.
The agency, in its infinite wisdom, has declared that when ERAM goes fully operational nationally there will be no backup whatsoever.
In Salt Lake right now, they no longer keep HOST going- so when ERAM failed, they had to use the much more limited system (EBUS, aka DARC). They ran on DARC for a couple of hours. The Command Center re-routed traffic around Salt Lake's airspace.
They came back up in an older build of ERAM. The one that failed actually had a significant number of fixes in it, but of course it could fail at random.
Worst thing of all is the political pressure being applied. At Salt Lake, they got a big time off award for the whole facility. Seattle has been running all but a few weeks of the same time as Salt Lake- no award. Seattle also has written up far more problem reports and the Seattle manager delayed implementation considerably while insisting the system had to pass basic functionality testing prior to going live.
Sour grapes from someone that didn't get the time off? Sure.
But it was also a clear message- play ball, or suffer.
Finally the FAA bigwigs had enough and took the decision out of the local manager's hands, and started making those decisions from DC.
This program has improved immensely, but it was a huge ripoff in the beginning. Engineers in the FAA knew, even when ERAM was supposedly passing tests, that it stunk. They quietly discussed its pending failure, but in Marion Blakey's FAA, nobody said anything to anyone outside the agency, and inside you had to be careful who you talked to as well.
Someday, this story should be told.
Ex-FAA Engineer (not verified)
December 6, 2011 - 7:46pm
Is it just me, or has the FAA gone steadily downhill since top management became infested with washed up ex-airline executives?
George (not verified)
December 11, 2011 - 9:07pm
The FAA had allowed itself to fall into the trap that Eisenhower had warned about 50 years earlier. Because of the lack of oversight and allowing Lockheed to pass along a faulty system we are at this point today. It is the correct decision to halt live testing in a rush to field the system. It was a critical mistake to exclude the controllers from the process and expect them to simply work around critical problems.
You could see this coming and couldn’t understand how the D.C. folks couldn’t see what they were doing was wrong. Our relationships with the contractors who build our air traffic systems are seriously flawed. When was the last time you had your car to a mechanic for repairs and agreed to pay cost, plus a fixed fee and if he put your car back together a bonus and if it started another bonus and if the problem was fixed yet another bonus. At the end of the day we did this too ourselves, the FAA is at fault. The folks in Washington better grow up and realize Lockheed is a for profit company, it operates in its own self interest. You have been fleeced as ATC-FREQ’s correctly points out you failed to heed the lessons of past failures, now you, very well may be history also.
In general most contracts with multiple delivers and thereby dates, like ERAM have performance incentives built in. It’s an archaic system left over from WWII on cost plus fixed fee contracts. The government needed material for the war effort so incentives were included in the contracts. The incentives portion as well as cost plus fixed fee still lives in government contracts today.
So as it pertains to ERAM; For B-level testing Lockheed was paid for a 60 percent pass rate. Since ERAM fell well below the 60 percent figure Lockheed and their lawyers got creative with the contract language. They were able to manipulate the pass rate by using a “NO TEST” NT category for test ERAM flat out failed. A second method employed was having tests categorized as “Working as Designed” WO so they actually were well below the pass rate of 60 percent but were paid in excess of a million dollars. You referred to people being under pressure to pass items that were obviously wrong, this is why. If they were one test case away from a million dollar performance bonus and you refused to sign certifying that the test meant a “Passed” criteria they found some else to sign. All of this had to be agreed to someone in the FAA program office, someone signed the change of criteria.
Another incentive area is IOC, when ZMP ran for 3 or 4 hours Lockheed was paid 1.2 million, that’s specially spelled out in the contract. Running from 1 am to 4 am on a Saturday morning with maybe a few hundred or less high altitude tracks isn’t a very high bar to have to clear for 1.2 million.
I’m not a contracts persons there maybe one that reads the blog but I have talked to enough to know there will be tremendous pressure for ERAM to be a “Go” whether it is ready or not.