Cybersecurity: How US utilities passed up chance to protect their networks

With America now trying to thwart a cyberattack on its natural gas industry, it is helpful to recall the hectic days after 9/11, when industry scientists raced to shield from potential terrorist cyberattacks hundreds of thousands of vulnerable devices that control vital valves and switches on America's gas pipelines, water plants, and power grid.

It was a race that seemed winnable. After five years of intense effort, a 35-member team of industrial-control-system wizards from the gas, water, and electric utilities industries had created a powerful new encryption system to shield substations, pipeline compressors, and other key infrastructure from cyberattack.

But just weeks before it was to be finalized in 2006, the federal funding plug was pulled on development of the encryption system, called AGA-12. Meanwhile vital industry backing from the American Gas Association and its partners at the electric power and water utility industries that once drove that funding had waned, say those who worked on the project. [Editor's note: The original version of this story mischaracterized the funding source for AGA-12 development.]

To this day, the cancelation of the project has called into question whether US utilities will, on their own, invest in measures necessary to protect their networks.

Tested at a Los Angeles water treatment plant, a gas utility in Chicago, and other locations, AGA-12 worked well. National labs verified it. Experts said it was good to go. Yet with 9/11 receding in memory, utility industry executives had begun worrying anew about the cost of deploying the system, former project participants say.

Today, six years after AGA-12 was aborted and 11 years after the World Trade Center attacks, the US natural gas industry is trying to thwart a real cyberattack campaign, according to the US Department of Homeland Security (DHS). Congress, meanwhile, is still debating whether voluntary or mandatory security standards are the best way to secure America's critical infrastructure.

All of which leaves researchers who helped develop AGA-12 frustrated and a little wistful about the digital shield that they say would have provided a badly needed layer of security – especially in light of a trend toward cyberattacks on critical infrastructure companies.

"Technically it was an excellent standard and we were almost done with it when the project was terminated," says William Rush, a now-retired scientist formerly with the Gas Technology Institute, who chaired the effort to create the AGA-12 standard. "One of the things I wake up in the middle of night and worry about is what to do if we've just been attacked. That's not the time to worry about it – now's the time."

AGA-12, he says, was designed to secure older industrial control system devices out in the field, many of which still today communicate by modem and phone line, radio, or even wireless signal, but were never designed with cybersecurity in mind and remain highly vulnerable today.

It's not clear that AGA-12 could have stopped the "spear-phishing" type of cyberattack now under way against the natural gas industry, experts say. But it could stop at least one kind: attacks directly on systems in the field of the kind DHS has highlighted in numerous studies and reports.

Installed in front of each vulnerable device would have been an AGA-12 gatekeeper, a sealed black box with a processor and cryptographic software inside, he explains. That "bump in the wire" would sift and decipher commands coming in from legitimate operators, but shield the vulnerable industrial control systems behind them from any false signals that might allow a hacker to take over.

"It was never intended to be a silver bullet," Dr. Rush says. "But it would definitely have provided quite a lot more protection for critical infrastructure like gas pipelines and the power grid than we have right now."

The reality of the cyberthreat was driven home in late March, when DHS issued the first of four confidential "alerts" warning of a cyberattack campaign against US natural gas pipeline companies' computer networks. Some researchers have linked the attack to a 2011 attack for which US officials blame China.

Those recent attacks follow a trend in which corporate and industrial networks belonging to critical infrastructure companies are seen to be a growing target. In April, the cybersecurity company McAfee and the Center for Strategic and International Studies (CSIS), a Washington think tank, found that 40 percent of electric utility company officials in 14 countries said their networks were under attack and more vulnerable than ever.

Meanwhile, in an election year, Congress and the Obama administration are wrangling over new cybersecurity standards for critical infrastructure companies – primarily whether they should be based on a voluntary or mandatory approach.

"The issue isn't a lack of standards," says James Lewis, director of the Technology and Public Policy Program at CSIS. "It's the lack of a business case for individual companies to spend for public safety. This [AGA-12 case] just confirms it. They know what to do to make things secure and have chosen not to do it for sound business reasons. A voluntary approach doesn't work."

At least six energy industry organizations that have developed voluntary cybersecurity standards for their industrial control systems would disagree. They include the North American Electric Reliability Corporation (NERC), International Electrotechnical Commission, American Petroleum Institute, and the AGA. But because the standards are voluntary or are "guidelines," it's unclear how widely they have been acted upon.

Asked if field devices have received added protections that supplanted the need for AGA-12, Jake Rubin, an AGA spokesman, says the AGA, federal government, and industry groups “have put cybersecurity guidelines in place that independent operators are using currently in the field.” However, he adds, “The ‘bump in the wire’ concept cannot be applied to all existing systems.”

"AGA members are committed to the safe and reliable delivery of clean natural gas to their customers at affordable and stable prices," says Mr. Rubin, an AGA spokesman in an e-mail response. "They must make decisions that balance these factors, with safety always being the top priority for America’s natural gas utilities.”

But other observers say that while some newer equipment with better security has been adopted in recent years, many of the same vulnerabilities remain because long-lived industrial control systems are rarely replaced if still functioning. Without a mandate, few companies will incur the cost to deploy enhanced security systems, they say.

“We found that the adoption of security measures in important civilian industries badly trailed the increase in threats over the last year,” Stewart Baker, a former DHS official who led the CSIS and McAfee study, said in a statement in April.

Critical infrastructure industry executives (oil, gas, electric power, water) made only modest progress over the past year in securing their networks, the survey found. In the energy sector, security technology adoption grew just one percentage point (to 51 percent) with oil and gas industries increasing by three percentage points (to 48 percent).

Even back in 2006 when memories of 9/11 were sharper, the business case for spending the money to become more secure just wasn't there, says Dennis Holstein, an independent researcher who helped write the AGA-12 implementation documents.

"What I think killed AGA-12 more than anything else was the cost of it," Holstein says. "It was a success. But nobody was willing to pay $500 for a bump in the wire solution even if it radically improved security. I haven't seen any deployment of it."

Protecting hundreds of thousands of miles of interstate gas pipelines, water supplies and even the power grid with the new encryption boxes was clearly a bottom-line decision, says John Kinast, a former senior engineer at the Gas Technology Institute, now retired, who was a primary researcher developing AGA-12.

"As time went on, and we got farther from 9/11, there was just this feeling from the industry side that, 'Well, gee – nobody's attacking us, so maybe it's not such an issue,' " he says in an interview. "But it's more than complacency. When you look at the cost-benefit and try to formulate a payback for a bump in the cord – for something that hasn't happened yet – it's just tough to make the case."

The urgency has reemerged at times. After revelations in the fall of 2010 that a digital weapon called Stuxnet had homed in on and wrecked centrifuges in Iran's nuclear facilities, it was clear to many that hypothetical threats to industrial control systems were for real – and many energy industry officials were alarmed.

"There was a burst of panic in the [gas industry] executive suites, and rightly so over Stuxnet, but at this point nothing has materialized," says Rush, the retired Gas Technology Institute scientist.

Now the gas pipeline industry is experiencing a cyberattack publicly identified in April by DHS, although it's still not clear to what degree the attacks are aimed at merely stealing information on corporate systems – or at mapping the control system vulnerabilities for operating natural gas pipelines.

“To our knowledge, the ‘cyberintrusions’ reported to DHS have had no impact on deliveries or the safety of the pipeline system," Don Santa, president and CEO of the Interstate Natural Gas Association of America, said in a statement May 8. Members of his association, which has its own detailed cybersecurity guidelines, operate 223,000 miles of the 319,000 miles of natural gas transmission pipelines in the US.

Even so, some say America needs to take more direct steps to protect aging critical infrastructure including, ironically enough, something like the AGA-12 standard.

Fortunately, about two years ago, the Institute of Electrical and Electronics Engineers (IEEE), a powerful body that sets standards for industrial electrical equipment, dusted off the AGA-12 protocol and renamed it the IEEE 1711-2010 preliminary standard. It is set to be finalized soon ­– about 11 years after research on it began.

But even now, selling a "bump in the wire" cybersecurity box remains a tough sales pitch for vendors pitching IEEE 1711-2010 boxes to gas, electric, and water companies that have old, insecure devices slathered across the American countryside.

"The vulnerabilities are still out there, but now we have the equipment to patch it," says Tien Van, president of Sequi, Inc., a Tustin, Calif., systems provider that began building IEEE 1711 equipment. "We have sold some, but not too many of these.… Companies still don't want to spend the money to fix this."