Pentagon is worrying about 'Terminator' coming true. Seriously.

The idea behind the Terminator films – specifically, that a Skynet-style military network becomes self-aware, sees humans as the enemy, and attacks – isn’t too far-fetched, one of the nation’s top military officers said this week.

Nor is that kind of autonomy the stuff of the distant future. “We’re a decade or so away from that capability,” said Gen. Paul Selva, vice chairman of the Joint Chiefs of Staff.

With such a sci-fi prospect looming, top military thinkers and ethicists are beginning to consider the practical consequences. But the more they do, the more it’s clear that there is considerable disagreement about just how much freedom to give machines to make their own decisions.

“We have to be very careful that we don’t design [autonomous] systems in a way that we can create a situation where those systems actually absolve humans of the decision” about whether or not to use force, General Selva said. “We could get dangerously close to that line, and we owe it to ourselves and to the people we serve to keep that a very bright line.”

At the same time, “The notion of a completely robotic system that can make a decision about whether or not to inflict harm on an adversary is here,” he added in remarks at the Center for Strategic and International Studies Monday. “It’s not terribly refined, not terribly good. But it’s here.”

This leaves top Pentagon officials to confront what they call the “Terminator conundrum,” and how to handle it.

The 'Russia' question 

The argument begins with an assertion: Adversaries such as Russia and China are going to build these fast-moving, fully-autonomous killing systems, so perhaps the Pentagon should design them, too – not to use them, top officials are quick to add, but to know how they work and how to counter them.

After all, they say, policymakers may need options, and it’s the job of the Pentagon to give them these options.

To this end, in a highly-anticipated report released in August, the Pentagon’s Defense Science Board urged military researchers to “accelerate its exploitation of autonomy” in order to allow them to “remain ahead of adversaries who also will exploit its operational benefits.”

This leaves opponents of autonomous drone systems wary. Many nongovernmental organizations have called for bans on developing killing machines that leave humans out of the loop.

But what is “meaningful human control”?

That concept hasn’t been well-defined, says Paul Scharre, director of the 20YY Future of Warfare Initiative at the Center for a New American Security.

What is clear is that humans can’t be involved solely in a “push button way,” Mr. Scharre says. Rather, the humans supervising these systems must be “cognitively engaged.”

Scharre points to two fratricides in 2003 caused when malfunctioning Patriot missile systems shot down a US F-16 fighter jet and a British Tornado over Iraq.

“One of the problems with the fratricides was that people weren’t exercising judgment. They were trusting in an automated system, and people weren’t monitoring it.” 

Humans are slow 

But the Pentagon’s debates get murkier from there. If an adversary were to develop an effective fully-automated system, it would likely react much faster than a US system that requires human checks and balances. In that scenario, the human checks could cost lives.

Frank Kendall, the undersecretary of Defense for Acquisition, Technology, and Logistics – essentially, the Pentagon’s top weapons buyer – has signaled that he differs from Selva. If people always have oversight of autonomous weapons, he says, that could put the US at a disadvantage.

“Even in a more conventional conflict, we’re quite careful about not killing innocent civilians,” Mr. Kendall noted at the Army Innovation Summit last month. “I don’t expect our adversaries to all behave that way, and the advantage you have if you don’t worry about that as much is you make decisions more quickly.”

After all, many weapons systems could be easily and effectively automated, including tanks that could sense incoming rounds and take out the source of them, he argued.

“It would take nothing to automate firing back, nothing,” Kendall told the audience according to the online publication Breaking Defense. “Others are going to do it. They are not going to be as constrained as we are, and we’re going to have a fundamental disadvantage if we don’t.”

But global bans on autonomous weapons are also problematic, Selva said. “It’s likely there will be violators.”

“In spite of the fact that we don’t approve of chemical or biological weapons, we know there are entities, both state and nonstates, that continue to pursue that capability,” he said.

Moreover, he noted that these questions put him – as a military man – in a difficult position. “My job as a military leader is to witness unspeakable violence on an enemy.... Our job is to defeat the enemy.”

The astonishing 'Go' experiment 

Removing humans from that decision to inflict violence could prove highly unpredictable in good and bad ways. That was brought out with dramatic effect at an event earlier this year pitting a Google-developed DeepMind machine against a top-level player of the complex game “Go.”

The machine, which humans had trained to learn, made a move that astonished Go commentators. “The move that the computer made was so unexpected and counterintuitive that it blew commentators away,” notes Scharre. “At first they thought it was a fake, and then it set in, the brilliance of the move.”

The point, Scharre says, is that “no matter how much testing is done, we’ll always see surprises when machines are placed into real-world environments.”

It’s particularly true in competitive environments – epitomized by war – where adversaries will try to hack, trick, or manipulate the system, he adds.

“Sometimes the surprises are good,” Scharre says, such as the Go move that was calculated by experts to be a “brilliant, beautiful” move that only 1 in 10,000 humans would have made.

In other cases, the surprises are unwelcome, highlighting the unexpected and tragic flaws that led to the Patriot air defense fratricides.

“Better testing and evaluation is good, but it can only take us so far. At a certain point, we will either have to decide to keep a human in the loop, even if only as a fail-safe,” he adds, “Or we will have to accept the risks that come with deploying these systems.”

Weighing the benefits  

“In some cases, the benefits of autonomy may outweigh the risks,” Scharre says.

Speed of decisionmaking is one example. Innovations such as self-driving cars also mark a “tremendous opportunity in the coming years to save tens of thousands of lives,” Scharre says.

The question, experts add, is whether these potential lives saved will outweigh potential – though likely fewer – lives lost if the automatic systems go awry.

For his part, Kendall imagines much bigger changes in weapons automation.

“We still send human beings carrying rifles down trails to find the enemy. We still do that. Why?” he wondered aloud. “I don’t think we have to do that anymore, but it is an enormous change of mind-set.”

“Autonomy is coming,” he added. “It’s coming at an exponential rate.”