The Value of a 3rd Party Pen Test…..or the lack of value

In my role I perform many functions, including penetration testing. I typically perform web application penetration testing but I occasionally perform a traditional penetration test as well. Also annually we bring in a 3rd party company to perform external penetration test against my organization Internet presence. There are two benefits for using a 3rd party, even when your organization has its own internal penetration testing team.

The first benefit is getting a new perspective. When perform a penetration test on the same network time after time it’s easy for the internal team to lose its perspective. For example if the internal team knows a certain network device only supports SSHv1 the internal team may stop testing the device. However an external team will continue testing the device, and perhaps locate a new vulnerability in SSHv1 for that device. The internal team stopped testing the device because they know it has an SSHv1 vulnerability that has not been mitigated and moves on to another target. This is occurring because the internal team has gotten tunnel vision.

The second benefit is showing the value of the internal team. Upon conclusion of the 3rd party test, there will be a report showing their results. Typically the results should show most of the same vulnerabilities, it may not show all of them and perhaps a few ones the internal team missed. Although the results are different it should be very similar to what the internal team results.

Now if the results from the 3rd party are vastly different from the internal team there is an obvious issue. This week I found myself in this position. The vendor did their “pen test” wrote a report and shipped to us.

After reviewing the report I found many glaring problems with it. First the complemented us on our quick response to their “pen test”, when in fact we did nothing on purpose. Our tools picked the test up and people started to respond like they would an in incident but I stopped them knowing it was a “pen test”. Second they complemented us on our amazing monitoring, which I have no idea how they determined this. The one though that through me was when the report stated they cracked some passwords. This bothered me because the scope for this test did not include any password cracking. So I immediately contacted the vendor to find out what passwords were compromised when the engineer told me he did not crack any passwords.

Now I have to question the entire report and the results in it. When I reviewed the results I was less than thrilled with what I found. The report appeared to take the results of a vulnerability scanner and put it in their report, called it a “pen test” and that was is it. Well the results were crap and the vendor missed a lot of known of vulnerabilities. When we scoped this pen test, we wanted the vendor to knock on our web apps and they claimed to have found only four web applications with vulnerabilities. Well that looks great but was far from the truth as I know for a fact they missed a lot more vulnerabilities on the web applications. When it came to vulnerabilities on the rest of the infrastructure they only claimed to find about six vulnerabilities, again missing a lot more vulnerabilities.

Luckily, my management knew about the vulnerabilities and where not happy with the results from this vendor. Another “feather” in my teams cap was showing is our tool set, process and procedures are working.

In this case the 3rd party vendor provided absolute no value to my organization and it appears my organization will be going through this process again. Typically this is not the case, but occasionally this type of thing will happen.

Some cool tools have been updated

Over the last few days a couple of cool of offensive security tools have been udpated. Both are peneatration testing tools, Metasploit and Samuari Web Testing Framework (WTF).

HD Moore released Metasploit version 3.5 today. There is over 600 exploits, over 300 auxliary modules and over 200 payloads in this release. This build also includes scriptjunkies java GUI. There are tons of other updates to this wonderful tool so I suggest you go to the metasploit website and check out the new version.

Kevin Johnson released a new version of Samurai Web Testing Framework (WTF). Samurai is a web application security testing liveCD that has a bunch of web app security testing tools. Another cool feature is the several vulnerable applications on the CD for learning and testing purposes. Go check it out. Here where you can find the details about WTF here.

Now that I am done with my malware presentation I will start I plan to get back to more blogging.

Presentation on Behavioral Analysis of Malware

I belong to DFW IT Security Professionals organization, if you are in the area check it out,  and volunteered to give a presentation on Malware Analysis. Since I wrote my GSEC gold on the subject, and perform the analysis often I like to think I have good knowledge on the subject. So on Oct 19th I will be presenting Behavioral Analysis of Malware.

This presentation is about performing behavioral malware analysis. I am covering the basics of malware analysis, why you perform the analysis and the types of analysis. I will go into the process that I have found works well for me. I discuss setting up a malware analysis lab, the tools to perform analysis, executing the malware, observing the malware and finally building the results of the analysis.

With that being said I hope of the next couple of months to go over parts of the presentation in much greater detail to help others successfully and safely perform malware analysis. I will send an update on how the presentation went and possibly post it on the site if there is interest.