Verizon DBIR 2016

I’m a few days late in posting this, I like to get it done on Monday. Maybe I’ll move that to Fridays from here on out, but I still intend to do it weekly. It has been helpful so far, I feel like I can talk confidently about the things I have posted on.

So Verizon released their Data Breach Investigation Report for 2016, to much controversy. Of concern to me, of course, is the Vulnerability section, which has been the source of the most controversy. It was a product of Kenna Security, spearheaded by Michael Roytman, a well known data scientist. So this thing has some credentials, and it is pitched specifically as being actionable for vulnerability management personnel.

Really this whole thing frames another semi-popular topic over the week, which was impostor syndrome. Ben Hughes wrote a blog post about this, and it is something I have struggled with for a while. In a field with so many talented people it is easy to forget that the most visible people are also the most extremely talented in the pool, and that even they have weaknesses or strengths. It is important to look at yourself with perspective, but also to look at others with perspective. If you see an amazing presentation at a conference, you’re seeing a point on a timeline that started years and years ago. It should be inspiring rather than making you feel like you’re not a part of the group. Which, you know, easier said than done.

But the DBIR situation really demonstrates to me how important this is to overcome. Verizon is Verizon, Roytman is very intelligent, and Kenna has done some great work. In the face of this it is tempting to look at their data points on vulnerabilities and assume that any issues you have with it are due to lack of experience or lack of data. It is important to understand the flaws when you see them. The sample is limited, even if it is a large sample. The results were not pruned, leaving many DoS results in the top vulnerabilities that just do not lend value. The methodology was disclosed, yet there are vulnerabilities on the list that could not possibly meet the requirements. The results themselves didn’t make sense, with specific vulnerabilities being called out that probably have never been exploited at all, let alone enough to demand a spot on a list such as this.

The value in this report is in the applicability. And the problem with the report, at least the vulnerability section, is that it has no application. I cannot in good conscience take these results and advise my engineers to prioritize remediation of the FREAK SSL vulnerabilities over newly released Microsoft patches, as Roytman suggests. Big picture stuff is great, but at the end of the day I have a network to help protect and following this advice would undermine those efforts. It is important to be critical, not disrespectful of the individual or the work, but still skeptical. And this is a point where you have to overcome the impostor syndrome paradigm and understand that you don’t have to be a renaissance hacker to realize that these results are sorely lacking in operational perspective. I can’t apply these, I can’t really glean much from them aside from making some assumptions about the data set they came from. This is an important lesson to keep in mind. No matter the source, be skeptical.

Rob Graham posted a synopsis just yesterday of how they came about this. He has an IDS background and breaks it down pretty simply. But the bottom line is that, as I said above and he said in his own blog post, the data is not actionable. This is the most research I’ve ever done into an industry report, and to see it so full of holes is distressing.

Last week of school, last final is today (*it was Monday, I am late) then I can refocus on more pertinent stuff. Next week I will write about Core Security’s vulnerability management maturity model I planned to do that this week but this whole DBIR thing seemed to butt right up against the other discussion about impostor syndrome, and this is an important thing to think about. I hope to keep this blog up and look back years from now at how much I have changed in this respect.