How often does someone come up to you and talk about non-functional testing? It is a catch all bucket for everything that isn’t a functional test case (doh!). This can’t be a thing! Yet until recently we had testers in our organisation that were described as non-functional testers working in a non-functional test team. This is because functional testers couldn’t do non-functional testing or so our supplier said. Actually that is a bit unfair, they had services which required specific people to undertake performance, operational acceptance testing and possibly elements of security testing.
In our waterfall delivery approach this was quite helpful to the planners. This suggested non-functional testing came later and planned as such. There was even phases called non-functional test phase which encompassed primarily performance, operational and implementation testing. We didn’t even need to think about what we were going to test in these phases till much later. Those pesky non-functional requirements for that later phase could wait till we had got the functional requirements out of the way. This led to a programme 1 year in not having thought or documented their non-functional requirements for the functionality written. We had one guy in the non-functional test team that would go round with a big non-functional checklist and belatedly get those projects to think about what they’d want to test.
When we started more agile ways of working, we started to get teams who felt that automation checking activity was the only required testing. We are doing TDD/BDD, why do we need to do any other testing. Our story and acceptance criteria has passed! Teams didn’t even know where to start. Today it is still a poor part of our lifecycle with some acceptance that we need to do device testing, browser testing, maybe some accessibility testing. However because the performance and different operational tests was done by other people our testers working in agile teams have been slow to look at operability, performance, security as part of their testing. Much of the time invested is still in functional testing but now using exploratory techniques. I am trying to avoid the use of manual testing but that is a whole different conversation.
For our peak sales period we still have a completely different team ensuring performance of the website. The same team benchmark’s performance on a regular basis as part of our release process. Although elements of this are worthwhile. Platform thinking would suggest a team helping other teams to improve performance is a good thing, especially where the platform and not specific features built within a single team are the overall performance goal.
Experience suggests things are changing, however there remains slow adoption. We have removed the separate resourcing arm for non-functional testing. We have people who specialise and can bring those skills to the team. Performance testing is still largely conducted in isolation in a single large performance environment which is often contended for use. We still lack testers in the team promoting operability (we lack people from operations in our teams also to do this, you could argue this is part of the problem).
The danger is that unless we demonstrate value in the team then why have testers as part of that team? Unless we can demonstrate that testing is more than just the completion of functional acceptance criteria on a story then we will lack ability to promote the need for testing as a whole and the addition of people in the team or supporting across more mature teams.
So how can we work to change things? Key is promotion of testing activity throughout the team (it isn’t just having a skill tester). Everyone has to have a responsibility. We have lot’s of teams now delivering so sharing what each other is doing is a key thing. The community aspect of testing and what teams are doing is important. Getting back to basic’s is also key. Agile Testing by Lisa Crispin uses basic agile quadrants as a way of describing the types of testing that need to be undertaken. In many ways working in a team is like starting your testing journey over, especially where testing in the past was part of a silo’ed phase that someone else did. Dan North uses similar quadrants as a way of educating people of all the different types of testing out there and encourages teams to explore whether they should really be thinking about them. This suggests where specialisms may be required within the team or to join the team for limited periods to support.
Lisa Crispin: Agile Quadrants, Part of her Agile Testing book
As we start to explore other types of testing and start to check this on every build we can start to look at data ahead of later performance testing and assess whether there is a risk or need to do more, in production like environments. Getting simple page speed checks, or function performance tests (API based or otherwise) in to build tests is a quick way of getting feedback and trending over time as to whether performance is an issue. It’s again important to note performance is one example but then accessibility, compatibility, resilience, alerting, logging are all additional types of testing that can be explored in a similar way. There are lot’s of tools out there to support you. When doing this it is essential to note that running an accessibility checker and reporting the massive list it comes back with isn’t the end. It’s about interpreting that data and extracting what really matters. Much of testing is just that analysing data and helping teams with feedback on that in terms of what is important. Not just raising a defect to say it didn’t pass the accessibility checker.
Dan North: Testing Corners, part of his Testing Faster class
By thinking about these things, testers begin to acquire new skills. That Operational tester or Performance tester didn’t magically learn to test those things, he learnt them over time and became specialised in that testing. It’s not a brick wall you can’t climb but one you should encourage to knock down. It is also important to note that this doesn’t mean everyone should be generalists but knowing about other types of testing is a starting point to being more enquistative in developing that knowledge and doing it yourself or in partnership with you team. So dont think that a type of testing is something done by a person specialising in that type of testing. Think of it as a learning opportunity where you may need advice but is something you can do within the team. It is also important not to stop there and acquire skills which sit outside testing. It might feel like this is all obvious stuff but I still come across this time and time again.