The Value of a 3rd Party Pen Test…..or the lack of value

In my role I perform many functions, including penetration testing. I typically perform web application penetration testing but I occasionally perform a traditional penetration test as well. Also annually we bring in a 3rd party company to perform external penetration test against my organization Internet presence. There are two benefits for using a 3rd party, even when your organization has its own internal penetration testing team.

The first benefit is getting a new perspective. When perform a penetration test on the same network time after time it’s easy for the internal team to lose its perspective. For example if the internal team knows a certain network device only supports SSHv1 the internal team may stop testing the device. However an external team will continue testing the device, and perhaps locate a new vulnerability in SSHv1 for that device. The internal team stopped testing the device because they know it has an SSHv1 vulnerability that has not been mitigated and moves on to another target. This is occurring because the internal team has gotten tunnel vision.

The second benefit is showing the value of the internal team. Upon conclusion of the 3rd party test, there will be a report showing their results. Typically the results should show most of the same vulnerabilities, it may not show all of them and perhaps a few ones the internal team missed. Although the results are different it should be very similar to what the internal team results.

Now if the results from the 3rd party are vastly different from the internal team there is an obvious issue. This week I found myself in this position. The vendor did their “pen test” wrote a report and shipped to us.

After reviewing the report I found many glaring problems with it. First the complemented us on our quick response to their “pen test”, when in fact we did nothing on purpose. Our tools picked the test up and people started to respond like they would an in incident but I stopped them knowing it was a “pen test”. Second they complemented us on our amazing monitoring, which I have no idea how they determined this. The one though that through me was when the report stated they cracked some passwords. This bothered me because the scope for this test did not include any password cracking. So I immediately contacted the vendor to find out what passwords were compromised when the engineer told me he did not crack any passwords.

Now I have to question the entire report and the results in it. When I reviewed the results I was less than thrilled with what I found. The report appeared to take the results of a vulnerability scanner and put it in their report, called it a “pen test” and that was is it. Well the results were crap and the vendor missed a lot of known of vulnerabilities. When we scoped this pen test, we wanted the vendor to knock on our web apps and they claimed to have found only four web applications with vulnerabilities. Well that looks great but was far from the truth as I know for a fact they missed a lot more vulnerabilities on the web applications. When it came to vulnerabilities on the rest of the infrastructure they only claimed to find about six vulnerabilities, again missing a lot more vulnerabilities.

Luckily, my management knew about the vulnerabilities and where not happy with the results from this vendor. Another “feather” in my teams cap was showing is our tool set, process and procedures are working.

In this case the 3rd party vendor provided absolute no value to my organization and it appears my organization will be going through this process again. Typically this is not the case, but occasionally this type of thing will happen.