Article - Why penetration testing may not be a good thing

Author: Werner Schmidt, CISSP
Date: January 2009

I have a lot of real problems and concerns with penetration testing. Just to recap, because terms are used so wildly these days, penetration testing to me is referring to having an outside organization actively test and attempt to break into an organization. In other words, actively attempt to penetrate some key servers or retrieve sensitive data to prove that it can be done and then hopefully to fix or advise how to fix the hole.

Penetration testing seems to be misused as either a quick fix or confirmation that all is secure and sound. I have a lot of issues with this approach when it is the only a small piece of a security puzzle:

It seems silly and laughable as a first step
Beware of conflict of interest
Networks evolve in both approved and unapproved manners
Threats evolve continuously
Devices come and go from the network (either intentionally or with subversive intents)
What is a perimeter?
What about wireless networking?
Some security devices by changing fingerprint and dropping connections can make an insecure site appear secure
What about application weaknesses?
What about backdoors?
A good hacker will cover up the exposure
It doesn’t address the outbound traffic and control channel scenario (simple example GoToMyPC)
It doesn’t address the risk of roving PCs
It can’t address encrypted communications via outbound traffic
Basically it assumes inbound directed threats
Basically it assumes externally originated threats
You can accomplish a lot of the same at far less expense with passive penetration testing
You’re assuming the tester is of equal or better capability to the best hacker...
It doesn’t address the topology discussion (best practices)
It doesn’t tackle appropriate use
It doesn’t address compliance concerns
It doesn’t address tools and remediation practices
Penetration tests are done to a budget...
It should be used after everything else has been made secure

1) I always enjoy watching the show, name escapes me, where a couple decides they would like some experts to come to their home and attempt to break in to basically show them the weaknesses of the existing security, deploy a new solution, test it and voila they are now secure. It’s always amazing, they don’t have good locks, don’t have a simple alarm system, don’t practice logical security best practices, don’t ever layer security, and never a concept of penetration rings. All the folks have assets they couldn’t possibly imagine parting with or losing, agonize when shown what happened and then they’re all secure after a simple system is installed. It's just like a lot of businesses approach security out there and the appeal and fascination with penetration testing. They’re all amazed that their primitive and effective systems and practices have been breached and then feel secure after something else is added and the originally discovered hole is closed (maybe). Well, it’s not that easy. There’s no point in having an “authorized burglar” show your weaknesses if you leave your doors unsecured, show your valuables, and don’t use good habits. First, fix the obvious, change your ways, educate, embrace best practices and THEN validate all is well.

2) At our firm, we don’t do penetration testing. We think it’s a conflict of interest to provide security infrastructure and solutions and then test/validate their effectiveness. Accounting firms got in major trouble for this, so should security firms. Just like the TV show, they’re never broken into by the installer after they installed their system, yet any good burglar could have breached and found the weaknesses in all those "improved" implementations. They also create a false sense of security and don’t get businesses thinking that the reality is they must assume and prepare for the worst, which is that a breach and possible loss will still occur.

3) Networks are always changing. Any approach to rely on a yearly, quarterly or whatever cycle review as the only audit is flawed. Do you check your ATM balance or wallet that infrequently? We’ve been into secure sites, not even on an audit and then pointed out and found a wireless installed just the day before that was in the process of being configured, but not yet finished, fully exposed, no security and on the network. That short but large exposure is important to note and resolve quickly, not months down the road. An approach should be used to be aware of the changing network landscape and make sure it’s still secure. Think of how many sites have been infected by guests that plugged into their LANs and infected their network and all the recent flurry of articles about P2P (Peer-toPeer) related breaches. What if a thief had been more discreet and created a backdoor, is an annual pen test really good enough?

4) Threats are changing daily, hourly and by the minute. While a simple vulnerability scan may show all is well (very unlikely), the next day a new threat arrives and it can be exploited even though there was no change in the underlying AAA. Networks and security threats are dynamic. Security must be reviewed and maintained on a frequent basis. It’s really about risk management. What are the key assets, what are their services, what are their vulnerabilities and how do other layers like firewalls, application firewalls, etc. contribute to managing the risk? Has the security posture improved since the last audit and is it trending in the right direction and in an appropriate amount of time?

5) Pen tests are done during a certain window of time. The assumption is that the exploit is present during a test. That may not be true... Keep in mind that lack of exploits is not any kind of indication about lack of existing backdoors.

6) Pen tests typically test from the outside to the inside. In other words from the Internet to the target site. This assumes a defined perimeter or edge. In a world of multiple VPNs, wireless, various control channel techniques, it’s a rather limited view of the world. VPNs are a major security threat and sorely misunderstood. So a firm trusts their business partner, how many trusts does that trusted firm have that you don’t know about?

7) All tests are done on the assumption of the Internet as the medium of attack. This doesn’t address other forms of entry whether it be physical or wireless. Fact is that any threat vector should be considered.

8) Some security devices and systems can make life harder for penetration testing. These include ability to randomize electronic fingerprints of applications, resetting connections, temporarily suspending activity/access, etc. These alone are not entirely effective, but may make an otherwise insecure device or application seem secure. However, by using various recon, trickling attacks in other data flak and other techniques they may not really be secure.

9) While pen testing can test for some application weaknesses, it generally doesn’t know about other application weaknesses and implementation flaws that may be present. This might be unauthorized users, incorrectly set application settings, programming flaws, etc. This doesn’t address application change control either.

10) People really don’t grasp the power of backdoors... All security seems to be focused on detecting an attack in progress (very small window of time...) or looking for an exploit that a future attack could use. Any good approach would try to subtly do recon over a period of time, note exploits and weaknesses, then wait for a very long time, exploit the weakness and silently install a backdoor. After a system has been breached, if it has been done well, most security detection measures become completely ineffective. This is a MAJOR problem with pen testing. Pen testing assumes an exploit or hole has been left in place. It generally cannot find a system that has been breached, made secure and then allows a backdoor control channel. To find this, and good luck, requires looking at traffic originating from within and going out. I hate pen testing for this, because users assume they can have weak security, pass a pen test and all is good. NO, strong security must be in place from the beginning! Most security approaches assume systems have not yet been breached, this is a major shortcoming of the mindset out there. If your home is secure, but someone installed taps, nanny cams, microphones and other snooping devices before you were secure, how do you feel now? Think about the stories about insecure US embassies including one that had to be demolished and started over from the ground up due to the numerous threats already in the walls. Secure from new threats? What about the existing threats from before you became secure?

11) Related to above... A good hacker will not leave an exploit open. A good hacker will not recon, probe and exploit all at once. A good hacker will not open new ports to communicate, unless use an existing approved one (think DNS, ping and other approved ports for transport. Basically a good hacker will cover up the trail. Pen testing is looking for a hole that has not been closed. A good hacker should close all exploits on a desired target system and not leave the glaring deficiency in place for a pen test to discover. The exploit is no longer needed after a breach has been successful because a communications and control channel would have been created and established.

12) So many people use GoToMyPC, LogMeIn, and things of that nature and so many folks don’t understand how it works and can bypass most firewall and similar security approaches. Realize that GoToMyPC is a well known and authorized program versus something else that’s the same but unknown. A pen test assumes traffic is initiated from the outside to the inside. GoToMyPC is a great example of how to very simply get around that, NO inbound traffic is required. It creates a backchannel that initiates from the corporate PC/network, tunnels out over an authorized http port 80, uses encryption and then communicates to what is essentially a proxy service. A user connects to that proxy service outside the corporate network and then issues commands over that backchannel back to the corporate system. It’s an ideal model how a hack can easily be done and completely avert pen testing. I’ve focused on GoToMyPC as a simple example, but in reality any outbound observed actual traffic patterns have to be considered. If a key server, perhaps a credit card server is making a lot of connections to a country like Russia or China and you are a US only operation, that should warrant EXTREME attention, much more so than any penetration testing results. You can’t just look at the inbound, you HAVE to observe and analyze outbound traffic. Don't just assume traffic has to go to a suspect country either, just use a hostage bot or proxy in the US and then have that route the traffic to the destination country.

13) Roving PCs are usually in the field. Perhaps they come into the office, perhaps they VPN in, but any threat that the PC has might only be present while they’re on the corporate network. While they’re roving they don’t have the protection of the perimeter defenses and detection mechanisms. All you need to get into a corporate network is access to one device or one account that has access to the internal network. OK, that me be an exaggeration, but you get the idea. All you need is one way in to exploit. All you need to break into a home is one entry point. Pen testing isn’t always aware of all the ways in. I used to run password crackers against large systems I supported (looking for weak passwords) and I could easily guess a few percent. A few percent of thousands is a lot and you only need one.

14) Encryption is going to be the next nightmare in security. While the traffic is still present, what’s unknown is the content and therefore intent of the traffic. This is a whole separate topic discussion. It is related to GoToMyPC and things like that. We cannot just look for connections and sessions, we have to understand and validate the payload contents. Think of a physical checkpoint inspection (I live in California near San Diego) that never checks the contents of a vehicle or sea port security that never checks containers. Same thing, you have to inspect payload for threats. Encryption thwarts that inspection. Encryption does not have to use special ports like port 443. If I wanted to avoid detection, I would use port 80 and then encrypt the payload, simple systems look for traffic marked as encrypted by using 443, again a naive assumption.

15) Already discussed this in above items, but pen testing assumes a threat is done from the outside in, very flawed in principal. Internal systems have to be considered as threat vectors. Already mentioned too, but a threat originating from the outside does not have to flow from outside in, backchannels allow threats from the outside to look like outbound sessions. Simple stated, we're looking in the wrong direction.

16) Pen testing assumes the threat is from an outsider individual or firm. This doesn’t address the insider (employee, contractor, etc.) threat or any of the various trusted business partner VPNs. Most threats come from the inside now.

17) Pen testing is all about show, just like the TV shows. Look what I can do... Well, passive pen testing can accomplish the same with much less flash and lot less cash. If I can find a known exploit that is in your organization and is exposed to the Internet, do I really have to enable that exploit, create a breach and prove it? That’s just extra expense. If a door has a weak lock at your home, does someone really have to break the door to prove it? Do you really have to break the pane of glass on a door to unlock it to prove that the weakness exists? In my opinion, that takes too much expense and time and possibly disruption to prove what is already obvious and should be addressed. Use resources to find all the possible problems, not just harp on one.

18) I’m personally not an excellent penetration tester. I also don’t break into homes for a living so I don’t know all the tricks that are used by those in the “profession” of thievery. My point is that just because a pen test shows all is well, it certainly doesn’t mean it is. I also don’t believe a lot of money should be spent until effective measures are in place, THEN bring in the experts to validate it. If it isn’t secure, have someone else remediate the weaknesses and then test again, perhaps by someone else.

19) Here’s the rub, over simplified, but all you need is one point of entry. Network topology often makes this very easy. If the topology isn’t sound, then one critical layer of defense is gone and now the endpoints (servers and desktops) have to provide all the security. Somewhat like having a home with a weak window point of entry, now the perimeter has been breached and your stuck with internal security and hiding of assets for protection (think cameras, safes, internal sensors, etc.). Not a very effective technique. A simple example is not having a DMZ. Just using one DMZ and not separating users from servers is another discussion, but if you don’t have a DMZ and use one to one NAT, then any breach into any of those NAT exposed servers gives you the keys to the kingdom. This is a little understood security concept of leapfrogging. Using one system to leapfrog into another system and so on. If a NAT device is in a trusted network and you breach it, from there you can reach into other systems and networks. Game over.

20) Pen testing is nice form a security standpoint, but it doesn’t address appropriate use. Fact is those who engage in visiting porn, gaming and hate crime sites are exposing their companies to risk. Then there is the whole risk discussion about appropriate use. This isn’t just an HR issue, but also public safety and business productivity. If you’re using IM in a police dispatch situation, it exposes a lot of risk to systems and humans.

21) Pen testing doesn’t address compliance concerns. If you are bound by compliance concerns such as SOX, HIPAA, GLBA, etc. pen testing may be a factor, but it doesn’t address all of it.

22) In my opinion, pen testing is about show and not a lot about educating the customer about best practices tools for detecting breaches and tools to remediate the threats. After all, the business model is to continue to find net new breaches each and every time to show their “value” to the customer. In the world of pen testing, there is an almost limitless supply, so you can keep a customer on the hook for a very long time by just exposing and fixing one threat at every audit interval. There is little interest in educating the customer to purchase tools and acquire knowledge to protect themselves and not use the pen tester’s services in the future.

23) Here’s a harsh reality, penetration tests are done to a budget. Just like TV, the point is to find something, anything and prove that there was a weakness that could be exploited. Bow, listen to the applause and leave while you’re ahead. While scripts can be used to show all the weaknesses, the goal isn’t to maximize the number of vulnerabilities that can be found, it’s about finding something, anything and then perhaps charging to fix it up. After a single vector has been found, it’s rare to continue and find/show/prove all the weaknesses. This is business, it’s about doing a task within a budget that a customer can tolerate. Customers might not be mentally prepared for the reality of just how many weaknesses could be found and repaired if budget were not a concern.

24) Hopefully I made my point by now, penetration testing is not and should not be the first and only step in a security plan. If best practices have been used, if security is a mindset, if appropriate use and outbound threats have been addressed, if appropriate monitoring and remediation in place, then consider pen testing as an additional security measure.

Read the rest of the articles. Please contact us for more information!