The penetration testing industry is in a definitional crisis. The “puppy mill” problem has been discussed before. The industry’s finest pundits talk about what penetration testing ought to be, but most of the pen- testing worker class just has to live with those ideas occupying dreamland, and gut through something that doesn’t always feel right. I am lucky enough to live in dreamland…

I had a fascinating discussion with a colleague recently. The question at the center of our talk was how one can tell the difference between a penetration test and a vulnerability assessment. I have long felt that if mention of a vulnerability scanner appears in your first paragraph of describing penetration testing, you don’t know what you’re talking about. It’s not a completely fair opinion, but it has some decent shock value.

But I kind of do feel that way. In my job, I live in what most people would consider an idyllic pen-testing wonderland. I get goal-directed pen-tests in hardened environments that force me to push the boundaries of my creativity. I have to write my own tools, strategize to overcome significant barriers, and I never have to compromise the integrity of my methodology for political or economic conerns. Others aren’t so lucky.

I see the difference when I get to do interviews. I’ve been interviewing people for penetration testing jobs now for about six years, and I’ve settled on several key questions to figure out where someone is.
One of those is, “What is your concept of the ideal internal penetration test?” I notice that most people I interview say something like this:

First, I run Nessus. I look through the high-risk vulnerabilities and figure out if I have exploits for any of them. I usually look for those that can be exploited with Metasploit. If I don’t, I check exploitdb and google search for exploit code. If the exploits work, I try to find domain administrator rights, and when I do I win!

I’m not going to say that this approach isn’t going to add value for a customer. But for an environment that has its general IT hygeine put together, it’s not going to pan out. And if this is your model, it means you won’t meet your success criteria as often as you could. What this is, in real life, is just taking a vulnerability assessment to the next level. It’s like 1.5x better than a vulnerability scan, and it’s not really penetration testing.

What I do on a penetration test is almost exactly the opposite. In fact, I enjoy looking back over a pen-test report and noting that not one vulnerability I found and exploited would appear in a vulnerability scan. Few of them show up in industry lists like the SANS Critical 20. In fact, it is my contention that at least 80% of the vulnerabilities I use literally cannot be detected by a scanner.

Let me give you a concrete example – I had an assessment not long ago where I plugged into the network and watched the broadcast traffic. I usually take 5-30 minutes just to watch the subnet to see what goes on. You learn a lot that way. One thing I noticed was that it was a big subnet. I didn’t even have a DHCP assignment yet, but I could see that this was bigger than a typical /24. All the normal Windows broadcast showed a bunch of end-user workstations.

But what stood out the most was that the hostnames looked eerily like usernames. There was a prefix and a hyphen and stuff, but basically the hostnames were something like “nyc-jsmith”. Hmm. I collected the NetBIOS traffic for awhile and put together a list of hostnames. A quick grep and awk through the PCAP and I had a nice list of usernames. DHCP. From domain advertisements, I knew where the domain controllers were. I picked a password with three character classes based on the season of the year and ran through my list. Bingo. Something like 5-7 accounts with a dumb password.

Authenticated access to the domain, time to loot file shares, and I’m off to the races. The assessment went on to accounts with some local admin here and there, a little lateral escalation, and control of the domain. My next phase is watching the users to learn how they behave – pretty soon I know where the crown jewels are, how the users get there, and I put together the credentials and network access I needed to get my success criteria.

Now I’m not sharing this story as some sort of chest-thumping display of prowess. All of my pen-tests go like this. And I think of myself as a journeyman pen-tester. There’s no 0-day magic… in fact, there’s not even any exploitation of “normal” vulnerabilities. No buffers get overflowed, and no SQLs get injected. But the take-home is that no vulnerability scanner could ever tell the customer what a realistic path from nothing to everything looks like in their environment.

The vulnerabilities I use all the time are bad passwords, domain trusts, gaps in firewall rules, split tunneling, pivoting, tunneling payloads through odd protocols, man-in-the-middle attacks, and evasion of defensive infrastructure. These can’t be scanned for (currently). And the fact that they lie dormant in all kinds of networks shows that IT personnel aren’t trained or positioned to realize the implications of all of these decisions. These vulnerabilities are the security decisions made by the users and architects of these environments.

And that’s value. You can patch everything you want, but if you have no way to reign in the risk from passwords your users choose, or if you don’t know how easy it is to chain together tenuous links through domain trusts and firewall rule exceptions, you might be a brain-dead easy breach just waiting to happen despite passing vulnerability assessments and “puppy mill” pen-tests.

And that’s the point. The definitional crisis in our industry threatens the convergence of pen-testing and vulnerability assessments. But if that occurs, the real harm is to the users, customers, and the critical, high-risk data we all have to protect. If your penetration test looks kind of like a Nessus scan, then you’re not really getting a penetration test. The value in a good pen-tester is that he is skilled in the arts of exploitation and post-exploitation, and he’s good at finding things.

So what should you do? What if you’re trapped in a puppy mill? First, follow Dave Kennedy’s advice from his presentation at DerbyCon last year – if you don’t want your job to be replaced by automation, do things that can’t be automated. Always go that extra mile, and find a way to add your expert interpretation of the results to guide a customer towards securing their environment.

Be that force for change, helping people understand that hacking isn’t about exploit code and patch management. It’s about noticing things in places where others don’t even see places. It’s about thinking backwards and wondering, “What if?” Children are better hackers than most adults, and a good hacker will still make mincemeat out of a place with stage-5 industrial maturity, strong patch management, and contracts with leading defensive control vendors.

And by all means, when you go to interview for that mid-tier or top-tier pen-testing opportunity that you’ve hoped for, don’t mention Nessus when asked what a pen-test is. Prove that you know it’s an art- form that requires creativity and that you, as a pen-tester, have an opportunity to uncover and help mitigate risks that could otherwise lie hidden for years. You, too, can actually win every time and really help people secure their stuff and keep the world a little safer.