A tester’s thoughts on Automation and AI part 2

One exciting possibility is to use AI to help with testing activities. This is called Cognitive QA and it can be used for a lot of different things. For instance, we can use Cognitive QA to tell us which test cases, we should run after a change. In that case it’s important that the system is transparent, so we can determine what criteria the AI uses to determine the best test cases. The answer will be different depending on criteria. Is the system looking at what functionality was changed? Is it looking at what part of the system is affected by the change? Is it looking at the risk of the change, and how difficult it is to make the change? Is it looking at the probability of bugs and if so, what criteria does it use to determine that? These are just some examples of the different types of criteria that the system can use to determine which tests cases it recommends running. The important thing is that we know the basis of the recommendation. Why was these test cases selected and not others?

One example that I heard during EuroSTAR was a system that used a picture to determine if it was of a wolf or a husky. They trained the system with a lot of pictures. After that they tried with ten different pictures and the system made the correct choice in nine of them. The problem was discovered then they analyzed the result more in detail and discovered that the system didn’t use the image of the animal at all. The choice if it was a wolf or a husky was based on the background!

If we’re using Cognitive QA, we need to test that they determine the result based on the correct criteria. And when we’re using it, we need to be able to determine what criteria it uses. What do you think?