Marking the transition to a new focus for QA and Testing

We’ve seen it coming for a while. The need to meet ever-demanding consumer expectations has been a steadily growing strategic imperative in recent years. This has now well and truly spilled over into the world of QA and Testing.

In our survey for the recently published 2018 World Quality Report, we observed the clear transition from technology validation to end-user satisfaction validation as the leading objective of QA and Testing strategies. This undoubtedly reflects the current obsession with where and how value is delivered to the customer. It’s no longer about whether we ‘can’ deliver IT, but whether that IT delivers value.

This finding is based on our survey of 1,700 CIOs and other senior technology professionals from across 10 different sectors and 32 countries.

The growth of AI

Perhaps unsurprisingly, the growth of Artificial Intelligence (AI) is a clear trend in this year’s survey findings. Some 57% of respondents said they had projects involving the use of AI for QA and Testing already in place or planned for the next 12 months. However, there is no finite way for testing AI at present and many organizations are experimenting with different approaches. What’s evident is that as the adoption of smart, connected technologies (think self-driving cars) continues to grow, AI will increasingly be used for analytics in testing.

Another interesting finding is that, despite the tacit understanding of the need to get new products to market as fast as possible, testing still doesn’t enable that speed – or what we refer to as Quality@Speed. A testing process that is ‘too slow’ is a top 3 challenge of Agile development, with 43% or respondents citing it as a challenge when developing applications. With Agile and DevOps now mainstream, and most companies using these approaches, this lack of speed will ultimately slow down ambitions.

Low levels of automation

A key barrier to speed-to-market is the horribly low level of test automation, representing just 14-18% of test activity. This is not due to a lack of the tools needed to automate, rather it’s about how automation is planned. It is often left to individual teams to implement their own automation technology, whereas what’s really needed is a strategic enterprise-wide test automation approach. With 61% of our respondents saying they have difficulties automating their QA and Testing processes because their applications change too much with every release, this clearly needs to be resolved.

Test environments and data are also among the culprits when it comes to achieving Quality@Speed. Currently, the lack of test environments and data is the number-one challenge our respondents face in applying testing to Agile development. While the immaturity of environments and test data remains a problem, there is hope on the horizon with growing use of such technologies as containerized test environments and BOTs. For example, 79% of respondents said they were currently using or planning to use BOTS for the set-up of test environments. At present, however, I believe that there is still a lack of cognisance around these technologies.

Embedding testers in Agile teams

We also observe the change in skills required to meet today’s QA and Testing needs. Testers must become part of Agile development teams, contributing a diverse range of skills throughout the development cycle. New testing technologies, such as AI, Analytics and IOT, demand more specialized skills in test teams and this will only increase in the coming years.

With QA and Testing becoming more integrated in Agile and DevOps teams, it is difficult to be wholly precise about the cost of test activities – it is often subsumed in an Agile budget. Nonetheless, the level of spending on QA and Testing appears to be stabilising. A clear trend is the continuing increase of the QA and Testing budget spent on hardware. Over the past five years, this has grown from a third of the spend to 44% this year.

Moving forward

A series of recommendations to take QA and Testing activity forward on its journey to validating end-user satisfaction, includes the following:

  • Focus on increasing the level of test automation
  • Adopt smart solutions and build an appropriate strategy for them
  • Move test environments and data provisioning from a siloed approach to an enterprise-wide focus
  • Build and elevate new QA and testing skills and embed them in Agile teams
  • Develop a strategy for testing AI – incorporating analytics, self-learning, and RPA.

The above findings and recommendations are comprehensively covered in the 2018 World Quality Report, which you can download below;



  • Mark Buenen
    Mark Buenen
    Head of Global Quality Engineering & Testing