Wednesday, May 4, 2022

Testing Mistakes that might be hard to spot

Testing Mistakes that might be hard to spot

Hundreds of test cases but not understanding the business use case
Unless you test the main use case, your 100s of test cases add no value. Do not go by numbers alone. Ask what those test cases cover. Is there a test case for every variation of test data and hence the inflated number? Even with the variation in test data, is it within the same equivalence class? Ask deeper questions.

Automating everyday but no one uses it
We must be doing good because our automation % is increasing. Do not fall for the trap. Again, ask the question - why are we automating, who is using it, how frequently? 
Check out the costs of automation.

Attributing a missed bug to lack of test case
Many teams add a test case as soon as a bug is missed. If your testing is 100% relying on test cases (which cannot be, even if you claim otherwise), the general tendency is to add a test case as soon as a bug is missed. How about asking the questions:
- Was it expected?
- Was it a known bug?
- Was it a result of the strategy used?
- What else could be missed?
- How will we capture those bugs?
- Why did these bugs come in the first place?
- Was there a possibility of catching them earlier - What would be the trade-off?

We will test everything every time
We can get into this situation if we don't understand the overall application model in depth. Agreed that there might be cases where every case is critical and thoroughly tested every time. Other than that, why not optimize, go through the impact analysis, analyze better?
Have you heard of RCRCRC mnemonic by Karen N Johnson? There are more here:

One dimensional coverage
Quality being multi-dimensional, it makes sense to think on all the perspectives and stakeholders. One shouldn't need a separate nudge to cover performance, security, accessibility, learnability, usability, compatibility and so on along with functionality. 

Incorrect usage of testing techniques
When was the last time you consciously thought about a testing technique while testing? Many are not even aware of testing techniques, forget about using them. Without knowing the techniques, either an incorrect usage or not using them is hard to find. 

How many of the above are you guilty of committing or ignoring?


Unknown said...

This issue is curated as a part of #56th Issue of Software Testing Notes Newsletter.