The flaws in penetration testing | Altaware, Inc. Cyber Security Blog

The flaws in penetration testing

Penetration testing is an important tool, but it's a tool I see being used incorrectly.

For the record, the best way to test and validate security is to do a penetration test. I think the full arsenal of sills and tools these people possess is impressive. As someone who likes old school (think cold war) spycraft, it's fascinating. Technology is fine, but human intel and good old social engineering are always my favorites. It's far more effective and harder to detect.

Penetration testing serves a variety of needs:
- It has a "shock and awe" impact on those who think they are secure and need their cages (and budgets) rattled to shake some sense into them
- It can validate the state or more likely lack of proper architecture
- It can detect flaws in implementation
- It can detect flaws in security training of all personnel
- It can show flaws in other areas (e.g. physical security)
- It can be enlightening as where sensitive data resides (any why is it there…?)
- It serves as another external set of eyes
- Even if you trust your security (big mistake), you still have to validate it

However, I have some concerns of how it is misused:
- If you haven't done the basic proper tasks, why are you wasting your money on it? (Unless you need to shock management into allocating funds)
- It's sometimes used to berate people/teams, the very ones that you need to keep you secure
- It's used to test defenses

My greater concern now though is it's used to test defenses. Lets face it, success is defined by at least one successful breach. Granted one is WAY too many, but lets also realize that 100% effective defense against any and all attacks is basically not going to happen. Statistics alone state that there exists at LEAST a 97% failure rate.

So, can it and should it be used to test defenses? Sure, but realize any good penetration tester with enough time and using their full arsenal ought to win… If they didn't, hire another one.

I've already said defense only is flawed from the start. A better focus now is to focus on a security methodology based upon defend, detect, respond and recover. In other words, while a penetration test will probably exceed in a glorious demonstration of the failure of existing defenses and training, can we instead use it to determine the ability of a team to defend but also then detect, respond and react to an attack? That would be a better test of the security infrastructure which includes the human component.

However, lets realize a major problem. Penetration testing is about finding a way in. It really isn't about assessing or determining if an existing problem already exists within the environment from a previous unauthorized successful attack. We need to shift our focus to detecting the presence of command control systems "dwelling" within the environment and our ability to notice it and how quickly we notice it. Even better yet the ability to determine what was accessed and for extra credit make sure any access was to encrypted data. We need to evolve beyond a defense only mentality in testing.