Glitches to riches: The hackers who make a killing off software flaws

A decade ago, anyone who uncovered a vital flaw in software might be rewarded with a pat on the back from their boss or a thankful e-mail from the software vendor. They’d earn bragging rights on Internet discussion boards and among their techie friends. But finding bugs rarely resulted in paydays.

Today is different. In the past decade, a growing, global marketplace for software vulnerabilities has transformed a talent for sniffing out security holes in software from a resume bullet point to something akin to Stephen Curry’s jump shot or Novak Djokovic’s serve: a rare skill that commands a high price. But with everything from software publishers to spy agencies and shadowy cyberarms dealers competing for prized vulnerabilities, experts warn that there are both risks and rewards for both society and the economy in what is quickly becoming a Gold Rush for the Digital Age.

The bug kings

"It’s like finding a gold nugget," says Mark Litchfield, a security researcher who has become one of the most successful and celebrated discoverers of software vulnerabilities. "Sometimes it’s like finding my own gold mine."

Mr. Litchfield hit pay dirt last September when he found 48 vulnerabilities affecting a leading website (Litchfield declined to describe the nature of the security holes, citing the terms of a nondisclosure agreement).

The collection of bugs netted him more than $63,000 in payouts through the company’s legitimate bug bounty program, with payments ranging from $50 for less serious holes to $15,000 each for critical and remotely exploitable vulnerabilities. 

The company paid those rewards through HackerOne, one of a handful of startups with online marketplaces that connect companies in search of security talent with independent researchers in search of payouts. 

HackerOne and similar sites act as middlemen, providing an easy-to-use platform for soliciting information on vulnerabilities and paying researchers for what they find, then taking a small cut. They also help foster a sense of rivalry among the research community, whose work often keeps them isolated. But, perhaps most notably, these new bug payment platforms are helping coalesce the growing cadre of vulnerability seekers.

Litchfield is the site's top-ranked vulnerability researcher and a fitting poster boy for a fast-evolving profession. With a close-cropped, military style haircut, he favors torn jeans and rock-n-roll T-shirts. He keeps himself well stocked with Marlboro Reds, an anathema in Silicon Valley where smoking is an express ticket to social Siberia. But Litchfield can afford to buck convention, having collected more than $300,000 in bounties through that firm since the company launched its bounty platform in 2013.

He’s no newbie, either. With his brother David, Mark started Next Generation Security from their home in Surrey, England, in 2001. NGS was an early security research and consulting firm, launching at a time when independent research to find software vulnerabilities was seen as meddlesome and the tech industry viewed security consultants with deep suspicion.

Back then, the Litchfield brothers and a handful of others started to make a name for themselves by pointing out security problems in software by such giants as Oracle and Microsoft. Over time, the Litchfield brothers expanded their research and developed specialized penetration testing tools to aid them in their work. Still, Mark Litchfield recalls, at the time, a bug hunting business was a hard sell to investors who could fund the business. They ended up selling the company to NCC Group in 2008 for a reported $10 million.

These days, Mark Litchfield prefers the desert of Las Vegas as his base of operations over global business hub of London or the tech epicenter San Francisco. It's a decision driven by practicality – Las Vegas is more affordable than either of those cities. Using Vegas as a home base also underscores Litchfield's core belief: Researchers like him are the “talent" in an industry that gives those skilled and crafty enough to sniff out the flaws in commercial technology the freedom to set their own terms.

So far, his theory is holding up. Litchfield’s ability to interrogate software applications and find his way around the protections built into them has netted him hundreds of thousands of dollars in bounty payments paid by firms including Yahoo, Shopify, Dropbox, Vimeo, and PayPal. And he isn't alone.

It wasn't always so. When the Litchfield brothers first started working on vulnerability research, their efforts didn't make them many friends among tech executives. Often, they would download free or demonstration versions of software from firms like Sybase, Oracle and IBM, then go to work breaking it.  

But companies had little experience working with independent researchers and often reacted with hostility when Litchfield and his brother, David, came to them with their findings. In just one example, the database security firm Sybase threatened to sue the brothers after they reported information on dozens of exploitable holes they discovered in the company's software in 2005.

These days, however, companies of all stripes see value in working with vulnerability researchers through company-sponsored bug bounty programs and bounty platforms. At the same time, startups such as HackerOne, SynAck, and Bugcrowd have made the job of creating a bounty program easier and helped those companies navigate the peculiarities of working with the vulnerability researcher community.

The website Bugsheet lists 369 bug bounty programs hosted by companies ranging from Adobe to Zynga. Less than half (153) offer paid bounties, with most (Including Adobe) simply rewarding researchers with public acknowledgment or swag. But that list is almost certainly too short, as it doesn't include the many, lucrative private bounty programs that sites like HackerOne host. 

HackerOne has 350 customers in total and hosts "hundreds of programs in invitation-only mode," according to Katie Moussouris, the chief policy officer at HackerOne. To date, the company's platform has reported more than 10,000 vulnerabilities to sponsor companies, she said.

Finding the flaws

Top bug hunters often describe an approach to finding vulnerabilities that is straightforward if not exactly regimented. “Most of my testing is manual,” Litchfield says. After being invited to a bounty program and asked to assess the security of a Web property, Litchfield says he often “jumps around from place to place” within a site until he finds features that he’s never seen before. That can happen even on well-established Web properties where one might think all the low hanging fruit had been picked.

“If you look at Yahoo Mail, that’s been looked at by thousands of pairs of eyes,” he said. “But the code changes all the time. New features are added. So I’ll go back to see what’s changed.” 

New functionality means new code, and new code invariably means vulnerabilities, Litchfield explains. But, just as often, it is legacy code that is often rife with exploitable holes. And for researchers working on bounty programs, holes mean money.  

The new platforms are also pulling in a talented new crop of researchers and helping fund new security-focused ventures such as Detectify, an automated testing platform cofounded by Frans Rosen, a researcher based in Sweden who is a relative newcomer to the hunting game.

Mr. Rosen has quickly risen among his peers, ranking second on HackerOne’s leader board and earning some $285,000 in bounties. That sum includes $150,000 in the past year alone – his best year yet. He won $25,000 for finding just one security hole during an invite-only hackathon at this year’s DEF CON hacking conference in Las Vegas.

Finding pay dirt is often a matter of intuition, he said. “Sometimes you can just feel that something on this site just feels vulnerable,” Rosen said. “I can’t put a finger on what it is, but if you’ve been testing thousands of platforms, you can just feel when something feels…not good."  

The fourth-highest ranked bounty hunter on HackerOne, North Carolina-based security researcher Sean "Meals" Melia, says he's earned close to $150,000 from bounties – and he just started participating in public bounty programs last December and does it as a side project while working a full-time job.

His big paydays have been met with healthy doses of disbelief from family and loved ones. Mr. Melia said he had to hire a tax expert to make sure he’s managing the tax implications of his bounty income. “Originally, I didn’t expect to make more than $1,000,” he said. “Now I have to talk to an accountant and get their advice," he said. 

While many top researchers are cashing in, bug hunting is inherently unpredictable and often time consuming and taxing work. “Sometimes you can find something in less than an hour. Other times it takes a couple days. Sometimes you might binge and report 20 things in one day,” says Melia. "People only develop applications with flaws,” he said. "The more applications that come out, the more flaws."

"He generally works on finding vulnerabilities after his day job is done – between midnight and 3 a.m. And, more often than not, those stints end in frustration. “So many people are doing the bounty programs, its hard to find things others are not finding." What's more, he said, companies fail to respond to his vulnerability reports, or fix a specific issue but fail to appreciate a more general condition that must be addressed to really solve the problem.

One of the biggest barriers for newcomers to vulnerability research are private bounty programs that are operated by firms such as HackerOne, Bugcrowd, and Synack. The programs are highly sought-after by experienced researchers, as they greatly increase their chances of success. But they also shut the vast majority of researchers out from the richest targets, something that Mark Litchfield worries will discourage others from joining the ranks. "I know plenty of researchers who say, ‘I’m not going to leave my day job until they change that.' "

He's hoping to change that through a new venture he's founded called Bug Bounty HQ – effectively a bounty startup seeded with bug bounties. The new site aims to treat researchers as the talent instead of just "the help," he said. The site will only award cash, not points for vulnerabilities discovered in invite-only private bounty programs, helping to level the playing field between veteran and newer researchers.

Bug, bounties, and bad guys

But while the global software vulnerability marketplace may be one in which “everyone’s making money,” as Litchfield puts it, is not immune to controversy. Not least among them: black and grey markets for exploitable software holes that are frequented by government intelligence services, wealthy corporations, and possibly even cybercriminals.

The greater acceptance of bounty programs and the emergence of new platforms like HackerOne and Bug Bounty HQ has merely greased the wheels of commerce – connecting talent with those willing to pay for it.

But like most marketplaces, the software vulnerability market is amoral. For every company willing to pay $15,000 through a bounty program for a remotely exploitable vulnerability in a common platform, there are individuals, governments, and the middlemen who serve them who might pay five or ten times as much – no questions asked.

The contrast between those two markets came into stark contrast this summer, when the US firm Zerodium offered $1 million bounties for working exploits of Apple's latest mobile operating system. The offer from the firm, which admits to having Western intelligence agencies among its customers, drew immediate criticism from civil rights and privacy advocates, who worried that any exploit sold to Zerodium might be used for unlawful surveillance or other means.

But Chaouki Bekrar, Zerodium's founder, dismisses those criticisms and says that his firm's $1 million bounty is simply the going rate for a working, remote exploit of what many consider one of the world's most hack-proof operating systems – work that will likely require the discovery and linking of multiple exploits in iOS or other components.

"Existing bug bounty programs offer lower rewards which are mostly adapted for proof-of-concept exploits on which a researcher has spent a few hours or days," he wrote in an e-mail. Zerodium and other firms like it are targeting the high end of the market: Weaponized exploits that are sophisticated and reliable, that might require many weeks of work to develop, he said. 

Global talent market

Despite flaws, vulnerability markets will tend to benefit society, rather than harm it, argue many experts and practitioners in the space.  "The proliferation of bug bounty programs is good for security," said Moussouris of HackerOne. "This is about the globalization and democratization of security talent."

While the market isn't perfect, Moussouris said that, over time, the industry will regulate itself – determining who the best players are, what market niches exist and what prices to pay. "It will be interesting to see how it all plays out. But you don’t have a true business or marketplaces if you don't have competition."

Editor's note: This story was updated after publication to correctly characterize Bug Bounty HQ's policy on awarding points to researchers who find software vulnerabilities.