, 26 April 2021

Throughout his career, Dan chose the path that he felt would best protect both people and the internet

Last week, we unexpectedly lost Dan Kaminsky, one of the security research world’s best known researchers. Dan was very highly regarded and very well-liked. He also did something in 2008 that truly helped save the Internet.

Dan was what we in the industry call a “penetration tester” — or “pen tester” — which is someone who tries to find problems so they can be fixed before malicious people find them and attack them. He was a good guy working tirelessly behind the scenes in a never ending race against the bad guys who wanted to commit crimes online. 

In 2008, Dan found a problem in the Domain Name System (DNS). Domain names are the addresses you type in when you want to access a website, like www.avast.com. Because computers don’t “speak” English — or Spanish or Chinese or Swahili  — DNS servers translate that human domain name into a computer name made up of numbers like 192.168.0.1 (called an IP address) to get to sites. And while many people have never heard of it, DNS is practically the glue that holds the entire internet together. It’s run by servers around the world and is so critical that some have special security protecting them. Dan would later be one of a very small handful of people trusted with the keys to some of these most critical servers.

The problem Dan found affected pretty much all DNS servers online at the time. We’re talking about tens of thousands of servers all over the world, running software from dozens of companies and projects. If someone malicious had found this problem and attacked it, the entire DNS system could have collapsed — and there wouldn’t be an easy way to recover from that. This issue was as serious as what you sometimes see in the movies and on TV shows. Except it was real.

Dan Kaminsky (Image credit: The New York Times)

After he found this problem, Dan had to decide what to do with this information. This is a question faced by every security researcher who uncovers a vulnerability. Dan decided to try and get this fixed by working confidentially with everyone who made DNS servers. It was a months-long process of research and coordination that involved hundreds, if not thousands, of people around the world working together to fix the problem before it could be attacked.

It was the largest coordinated response to a problem like this the world had seen thus far (and may well still be the largest). For months, software engineers at companies like Microsoft (where Dan and I were both working on this at the time) and Apple, organizations like BIND and openDNS, and others worked together to come up with a solution and put it into our products. It was an extremely complex endeavor and we had to work as fast as possible in case someone else found out about this or leaked the information before we fixed it.

On July 8, 2008, security coordinating organizations including CERT-CC in the United States and its peers around the world and makers of DNS server software like Microsoft, RedHat, IBM, Sun, Apple, among others, began to release their advisories and patches for this problem. To get an idea of how many organizations were involved, you can see that the CERT-CC advisory lists 91 vendors around the world that were affected.

Overall, it was a success. There was a leak of information days before the coordinated release, but in the end that didn’t derail the release or lead to attacks. In fact, it was so successful that the problem was never really effectively attacked. And, most importantly, the internet never crashed.

Dan would go on to share his findings with the security research world three weeks later at the Black Hat conference, where security researchers regularly share their findings. It was one of the most attended sessions in its history. You can hear Dan talk about it himself here.

I’ve left out some details that are specific to the security research world, such as the controversy about Dan’s decision to work confidentially rather than publicize the issue as soon as he found it. There are good arguments on both sides of this and it’s a question that always has (and always will) divide the security research world. The key thing on this point is that Dan took the path that he felt would best protect people and the internet, and he succeeded in that.

There’s a great short video of Dan explaining the problem himself here that goes more into both the history and technical details of this. It also gives a sense of his style and personality and helps illustrate why he was such a popular presenter at Black Hat.