Cheap Testing Explained
A post from my “In the Darkened Room” series, a personal look at cheap penetration testing and why it might, or might not, be a good idea.
As the owner of a penetration testing company I receive, almost daily, requests to “sharpen my pencil” or “give me your best price”. When I started back in 2010, I felt insulted but over time that feeling faded and now it is just a raised eyebrow. I only ever give my best price. I think it is fair and represents value for what we do.
With penetration testing, you are purchasing the skill of a professional tester, not a toolset or a license code. So I thought it might be useful to explore what happens when you want to pay less for a test.
All names are anonymised to protect people, but this is 100% accurate.
Demo Industries LLP (entirely made up) want to test a web application they have deployed within the Microsoft Azure environment. Their requirement was:
- Testing to be performed by a senior penetration test with at least 5 years of web application testing experience;
- Pentest to be performed by a CREST member company;
- The penetration tester must be CREST CRT;
- All testing to carried out during out of business hours (7pm to 6am);
- Test to follow the OWASP testing methodology;
- Testing company to have £10m in liability insurance; and
- Testing company and tester to be in the UK and all data to stay in the UK.
All of these requirements for me were easy and simple to meet. So on to the scope.
The application is deployed in Azure, and sits behind cloud flare. The architecture consists of 1 database, 1 back-end data processor, 2 API servers, 2 front-end servers and a load balancer.
There are five user levels:
- support; and
Testing should be performed from user levels 1 through to 4. All tests should be performed as each of these users.
Included in the test are two API functions and it all talks to an Azure database.
Our initial quote was for seven days of test time. It covers five days of continual testing and two days for documentation. We came to this time duration after a review by myself and one of Hedgehog’s other senior testers.
We always state that if we don’t use all the days, we will only bill for what we use. We produced a proposal that detailed what we would do during that time. We would cover all of the OWASP test points discussed in the scoping call, and it would all be to CREST standards.
The contact at Demo Industries LLP came back and asked us to look at the pricing and days because they had a quote from Competitor X, who had quoted only four days and were charging £350 a day.
Check 1. CREST testing requires a CREST member company. A check on the CREST website didn’t detail Competitor X. We called CREST and asked, and they confirmed no, they were not a member company.
Check 2. CREST testing requires a CREST Registered Tester. In our proposal, we detailed the available testers suitable for the project for the client to choose their Penetration Tester. All of them are CREST registered testers. In the proposal from Competitor X, no names or qualifications were detailed so it would be impossible to assure testing qualifications.
In the end Demo Industries LLP went with Competitor X. I was not too surprised to be honest, Demo Industries LLP seemed very cost focused. Being a local company, I knew the IT supplier that provided Demo Industries LLP with support on a day to day basis.
Imagine my surprise when, 3 months later, I get a phone call from the IT supplier. “Could you look at a pentest report for Demo Industries LLP please? The results look, well, wrong.” So we did.
What we saw was the direct output from OpenVAS, a free vulnerability scanning tool. There was no evidence of any actual penetration testing so what Demo Industries LLP purchased was in fact a very expensive vulnerability scan.
What to look for in a Proposal
Penetration Testing is an intricate art. There is an element of science to it, but real testing is artwork and can be a beautiful thing to behold.
The first thing to check is the understanding of your scope. Has the company understood the scope fully and does the proposal cover everything you set out in the scope?
Who is the tester? Are they listed by name in the proposal? What are their qualifications and their work history? A one page mini-CV is always useful.
What methodology is being used? What phases of test are there?
How often will you receive communications from the tester/test team?
Reporting / Documentation
When can you expect to receive the report? What other documentation will be included? Is there a test narrative? Will you get Proof of Concept code? Will you receive a .zip file with all the evidence and system outputs?
These outputs are extremely valuable. Most important through is a narrative. The narrative will prove or disprove that the pentest was done with nothing more than a few scripts and tools. You can also give the narrative to any other qualified seasoned penetration tester and they should be able to validate the results 100% from the narrative.