In testing, think – risk analysis and not shapes!

When it comes to testing software projects, the industry is obsessed with shapes. The Testing Pyramid is the most recognisable of these testing shapes. It describes how test suites should follow an ideal ratio of test types, typically, many unit tests, some integration tests, and a few end-to-end or user interface (UI) tests.

The core idea behind this is a good one. Unit tests are typically easier to develop and maintain than integration tests. In addition, UI tests are notoriously difficult to manage as they often test too much. So better to spend our time covering risks using easy to develop tests that are tightly scoped and provide fast feedback.

However, the theory does not always match reality. Pressures to deliver to deadlines, technical & testing expertise or testing culture problems can lead to an ice cream cone shape instead. The whole pyramid turns upside down as the team does what it can to address risk using known methods.

Source: Medium

Worse, multiple shapes exist which extol a different test suite ratio with its own set of justifications. The advice changes, the shapes morph. Suddenly we have many different shapes which conflict with each other! My personal favourite is the testing trophy. The name suggests success from the very start.

Source: Twitter

Today, we discuss:

• What is causing all these new shapes to be produced in the first place?
• Why does the advice change?
• What shape is the best to follow?

Risk > pyramids

That last one is a trick question. Projects have unique risks, and by testing the hope is to understand more about these risks and ensure, when possible, mitigation. So, there can be no perfect shape. Shapes are created to try to address the different common sets of risks that exist in software projects. But instead of relying on the latest shape to guide your testing effort, I want you to think instead of risk.

Risk is the key. Understanding the risks that exist on your project and adapting the ratio of your tests accordingly. Responding to the specific needs of your project. For example, a Microservice Architecture will have more integration points than a Monolith Architecture. But each service should have tightly scoped logic. In this case, more risk is likely to exist in integrating services than the services themselves. Having many integration (or Pact.io) tests would better address this risk than limiting the numbers because a pyramid tells you to.

Understanding risk as a team with risk mapping

If your team wants to move beyond pyramids and start thinking more deeply about how testing should be structured on projects, you will need to understand risk. One way to do this is to have a risk mapping workshop.

Either in-person or remotely, gather your team. Show them your entire system architecture in a diagram. Going around the diagram each person should leave notes on areas that represent risks. Keep going until everyone is finished and cannot think of any more to highlight. This is where you really start to visualise the risks.

Step back from the diagram and look at all the notes left. What does the picture look like? Are there areas where notes left by the team seem to cluster together? This could highlight a risky area in your systems architecture. An area needing investigation and potentially additional testing.

Let’s say your team works on a project with a heavy emphasis on User Experience (UX) on a web-based UI. Upon completing a risk mapping workshop, it becomes clear the team is concerned about this UI. Therefore, they would like to focus testing efforts on the UI to minimise the risk. Investigation reveals the best way to cover this UI risk is with automated UI tests. The pyramid would tell you this is to be avoided. But your team is now better informed of the risk on this project. They went ahead and completed a risk mapping workshop and now know that the UI needs to be covered.

Tailor your shape and test suite ratio to risk

The example above shows us why so many shapes exist. We are all constantly adjusting our test suites to better address the risks of projects. After our adjustments are completed, we create a diagram naming it, a new testing shape is born. But we do not need new testing shapes. Our effort would be better spent investigating new ways to identify project risks. Training our teams to be pragmatic in addressing them. Developing the wisdom to know when we should diverge from the pyramid.

By analysing risk your team will be more confident that these risks have been addressed by your test suite. Increasing the likelihood that it is catching issues before they happen. Your teams can do this by using risk mapping techniques to discover where project risks exist and help start a conversation on how to address them to help improve the likelihood of delivering quality. This will be much more useful than a debate on shape!

Learn more

• Watch this excellent talk on Testability by Richard Bradshaw at the Ministry of Testing
• Read Testing is Good, Pyramids are Bad and Ice Cream Cones are Worse

***

A bit about the author

Thomas Shipley is a Delivery Consultant at GlobalLogic. He mentors, pairs, persuades and promotes shared quality ownership in teams. He has applied these ideas in multiple sectors including Financial Services, Retail, Gaming and Government within well-known organizations such as HSBC, John Lewis, Microsoft and HMRC. You can find out more at his blog: tomdriven.dev.