The game of right questions

You always say that testing is all about asking questions. 

How do I know if I am asking the right questions?

Many testers have asked me the above question. Recently in my 1-1 coaching too, this question was raised. I thought it is now time for writing the detailed answer. 

Testing is all about asking questions about the product and evaluating the answers to see if there is a problem or not. Testing is also a sampling activity. Due to the limited amount of time, effort, it will be great if the right questions are asked early.  The problem comes when we need to figure out which are the right questions and which are not. It is similar to asking how do I find the critical bugs first compared to the minor ones. The answer to both the questions is similar - CONTEXT.

The more we know about the Context, our questions and our test ideas have a greater chance of being more pointed and useful. Let us work through an example. 

Let us consider two systems at work.

A web application that acts as the dashboard for the temple administration to view the darshan slots and count of devotees. Mobile application for the devotees to book the slots for the darshan. 

Though we can start from anywhere, I would like to focus on the below questions for a start:

Mission: 

Why are we testing? 

Is it a new release or upgrade? 

Based on the answers, your questions should change. 

If it is a new release, more focus is on what the customer has been promised. 

If it is an upgrade, the things that are already released should not break at any cost. 

The focus shifts drastically at just one answer. 

Deadline: 

How much time do we have for the release? 

How much time do we have for the testing? 

Is the development complete or still in progress? 

Users/Customers: 

Who are existing customers? 

Who will be our target customers? 

In this case, will the temple board be our first customer or individual temples manage the website on their own? 

Will devotees be only the senior citizens or will the apps be managed by agents who will help the devotees do the slot booking?

Key Features: 

Is there an existing list of features, documented in any form (Design docs, Code, Requirement documents)? 

Can we use the application to learn about the features? 

Are there test ids available?

Then start thinking about the four components of a test

Configuration

Which platforms will the applications support?

Supported OS Versions? 

Which are supposed to be supported?
How much of load can we expect? 

Which other components are in play?

How are the website and the mobile app connected? 

How frequently or how soon the data is synced?

Any specific time when the load of booking slots will be high?

Are there backup systems?

Where is the data saved? For how long?

How will we know if any of the systems go down? Is the scenario handled?

How is the temple onboarded?

How are the users onboarded?

Which other systems will take the load (SMS, notifications)?

Operation:

How would one book the slot?

How many slots can be booked in one shot?

How quickly can the slots be booked?

How long is the session active?

Any other modes other than online mode? (SMS?)

What about cancellations?

Is there a hold period?

Observation:

What happens to the UI before, during, after booking?

What gets printed in the logs?

What gets stored temporarily and permanently in the DB?

Are there other mediums of communication (SMS, Notifications, Emails)

Any download options (thereby invoking other systems and applications) of tickets?

Error messages and information messages throughout the booking process?

Evaluation:

What is the oracle here - the requirement document? the product owner? the product? similar products? user expectations?

How do we know that whatever we see on the dashboard is right?

Should we trust the website or app or something else?

Now, which of these questions are important and which of these are not?

We don't know till we ask these and get the answers. 

Good testers will not just go through some list of questions but modify the list on the fly based on the answers to previous questions or information gained midway. 

What do you think? How to win the game of right questions?

The more we know about the application, context, the right questions will appear quickly.

Meanwhile, join me for TestFlix - https://www.thetesttribe.com/testflix-2021/ by registering with your teams.

Leia Mais…

Four Quick Questions While Testing

This is a quick post in between a testing session. Right now I am testing www.ibibo.com and they have a 'Friend Suggest' feature where ibibo suggests friends for you.
Here is the screenshot from ibibo.com

Friend Suggest

I assume that friends are suggested based on some algorithm. The number of mutual friends is also highlighted.

I clicked on 'Add as a friend' against one of the profiles displayed. A popup was displayed.

Once you click on 'Add as Friend' link

While it is good to inform the user about an action he performed, as a user I was not able to perform any other action till I dismissed the popup by clicking on the OK or close button.

Compare that to what Facebook does. Even Facebook has the Friend Suggest feature by another name: 'People you may know'.

Facebook Friend Suggest

Once I clicked on 'Add Friend' link, the particular profile slowly disappeared after displaying 'Friend request sent'.
                                                          
Do you see the difference? Is there a problem here?

After this observation, I made some notes. Few questions which helped me find a problem were:

* What is the purpose of the feature?
* How will the users use it?
* Are there any problems in what you observe? Primarily any inconsistency or usability issues?
* Have you seen such a feature elsewhere? Comparable products heuristic?

Let us go through each of the questions in detail.
Purpose of the feature
As a tester, do you understand the purpose of the feature. What problem does the feature try to solve? As the product is a social networking site, more the friends, the better. The feature is trying to help you add friends. In the case of ibibo, does it really help?

Users using the feature
As a user, is it easy to understand the feature? Is there help available? Is it user-friendly? Can the user use all the feature's functionality? How is the first impression? Is the feature easily visible or hidden behind a lot of junk?

Usability Issues
This morning, I read the latest blog post by Michael Bolton - http://www.developsense.com/blog/2011/12/why-checking-is-not-enough/ on why checking is not enough.
I am not sure how a check might find the level of user's frustration or happiness.
When I was not able to click on any other link till I dismissed the confirmation popup, I was slightly frustrated.
Confusion, frustration, delight and other emotions - how will a check handle it?
As a tester, are you aware of your emotions when you test, after the tests?

Comparable with similar products heuristic
Finally, as a tester make use of the similar products heuristic if given a chance. Just as I write this post, I hear the dialogue from 'First Blood II' - Mind is the best weapon.  So true.

Maybe, next time too - I will use the four questions as a quick heuristic.
What is the purpose, who will use it, any usability issues, how the competitor handles it?

Leia Mais…

Do you TRUST the fix?

Something interesting happened when I was testing with my team.

One of the team members had logged a critical issue.

It was a bit complicated too. It involved a lot of parameters to consider.

After a week, the issue was fixed and the team member who logged the issue was verifying the fix.

After quite a bit of regression testing, I heard an interesting comment:

"Amazing!!! I'm surprised how could the programmer fix this issue without any side-effects"

Hmmm, it so happens that we expect the defects to be fixed.

And when they are fixed, we start questioning.

It is good to question but questioning everything.... being so skeptical... I don't know.

Maybe it is good some times but not always.

Do you TRUST your programmer and his fix?

Let me know.

Leia Mais…