No testcases for the scenario. What do I do?

One of my seniors is a big fan of Silk, Selenium, Loadrunner, TestComplete and any other commercial tool available in the market. He is very good at automating few tests depending on the product. My team lead had no tasks for me. So, he asked me to learn Selenium from the senior. My senior demonstrated some basic commands of Selenium and asked me to play for few hours. Now the next question was: 'Which application to use?'

There were two choices presented to me:
a. An application yet to be released to the market and still being tested.
b. An application released to the market after being tested by the team reporting to the senior.

I chose the second application. My senior told me that I'd not find any bugs as it was tested by his team. I took this as a challenge and explored the application.
In twenty five minutes, I found two inconsistencies. The severity of the issues was high too. With a big smile on my face, I called my senior and showed him the two issues. He immediately called the tester who tested the feature.

On being asked how the issues were not found, the answer given by the tester shocked me. His answer was: 'We do not have test cases for those scenarios'

Few strange lessons I learnt:

1. Some testers fail to test beyond the testcases. Should I call that testing?
2. Testcases give a false sense of security to some management.
3. Forcing a tester to learn any automation tool might not be a good idea in the long run.
4. A new pair of eyes - A new test idea - might lead to discovery of a new issue.
5. Its not a good idea to criticize or interrogate your team member in public.
6. I took it as a challenge to find bugs. Did the challenge attitude help me find those issues or the challenge had no effect?

What would be your answers to the question:
'Why did you miss those issues?'

Did I tell you that the tester added the two test ideas as testcases later?

Leia Mais…

A five minute conversation with a programmer

This is a post highlighting the conversation I had with one of my programmers.


Programmer (P): Hi Ajay! I need your help in reproducing the defect #abcdef

Defect #abcdef:
Step 1: ...
Step 2: Enter a value 50 in the text field.
Step3: ...

Ajay (A): Sure, How can I help you?

P: I'm unable to reproduce the defect.

A: Which OS are you trying it on? I had logged it on Win 2003.
P: Yes, I know that. It would be easier if I could reproduce that on Win XP.
A: (he he smiles) OK, I'll reproduce the defect first on Win 2003 and then we could try on Win XP also.
P: OK, great.

A: Step1, Step2, Step3 and here it is, REPRODUCIBLE!!!
P: Ok, let me try it. Step1, Step2 and Step3 and ??? Where's the defect?
A: Oh!!! Let me try again. Step1, Step2 and Step3 and again REPRODUCIBLE!!!
P: (Smiles) Step1, Step2 and Step3 : NOT REPRODUCIBLE

Silence for few seconds.

A: (thinking what could be different) Hmmm, maybe the speed with which I execute the steps is different from your speed of execution.
P: Maybe. Very little chance of that happening.
A: OK, let me try consecutive times. Step1, 2, 3: REPRODUCIBLE. Step1, 2, 3: REPRODUCIBLE.
P: Oh!!! How do YOU reproduce that Man!!! See, Step1, 2, 3: NOT REPRODUCIBLE.

P: Maybe you are pressing 'Enter' after entering the number.
A: Hmmm, I'm not pressing Enter key.
P: Then maybe single click causes this problem. Or maybe double click to select the field before entering the number.

A: (thinking WOW, so many factors!!! Let him go on)
P: Or this might happen if you select the text by 'Click and Drag' using mouse
P: OK, I'll look into this issue. Thanks for your time.
---------- END OF CONVERSATION ---------

This particular conversation refreshed my memory of how many different factors affect a single entry in a text field.

> Operating System
> Response time
> If the focus is on the field or not
> After entering the number, did the user press Enter or Tab?
> Did the user double click on the field to select the default text?
> Did the user delete the text before typing in the new text?
> Did he press 'Delete' or 'Backspace'

I'm sure there are many more factors related to the single entry in the text field.

The point I want to highlight here is "How useful is it to have a conversation with a programmer about the product?"

In my case, it was useful. Have you experienced such an interaction?
Do let me know.

Please feel free to share such experiences(Good & Bad).

Leia Mais…

Online or Offline??? Gmail or Gtalk doesn't matter

This is a strange behavior I observed while I was chatting online with my friend.

Application Used: Gtalk and Gmail chat

Reproducibility: 100%

Machine specific: No

---------------------------------------------------------------------------------------------------

I expect people chatting with me to reply within 5-10 seconds after I send them a message.

I also promise to reply instantly unless I'm not infront of my monitor.

It so happened that I sent my friend a message and I did not get any reply for a minute.

I got irritated by him not responding and I started banging the keys on my keyboard.

Infact I banged only one key - the DOT '.'

I pressed the '.' (DOT without quotes) and pressed enter key.

This continued for 15-20 times continuously.

Then these messages were displayed on Gmail chat window(pop up) and Gtalk application respectively.

As you can see on both the screenshots, the user is still Online indicated by the Green dot against the name.

And the message reads: "The user did not receive your chat" or "The user's email is offline and can't receive messages right now"

I liked this bug... Use of emotions helped me find this bug.

I don't really care if this is fixed or not but the behavior definitely brought a smile on my face.

:)

Leia Mais…

Simple Things Do Matter :)

I had to check a fix provided by a programmer to solve an issue.

I had to replace two files- One an .exe and the other .ini file.

Once I replaced the files, changed the contents of the files to suit my test and went through the steps to reproduce the issue, the issue was still reproducible.

So, I sent an email to the programmer that the issue is not resolved and the fix does not solve the issue. He wanted to have a look at my machine and try the scenario once.

Before handing over the control to the programmer, I copied the two files which were part of the test. This way I can compare the files after the programmer has completed his test.

The programmer was in Germany and was accessing the machine. I could not see what actions he was carrying out on the machine.

After ten minutes, I got a reply to that email that the issue is solved. Please re-test to confirm.

I was surprised and then along with my manager wanted to carry out the test. Before conducting the test, I wanted to compare the current file(after programmer conducted the test) and the file used by me to conduct the initial test.

And surprisingly, the contents were slightly different.

The initial file had 'Timer: 5:00:00 PM' and the new file had 'Time: 5:00:00 PM'.

Also, the programmer asked me to include an additional step so that the issue is resolved.

The additional step was to exit the service and run the .exe so that it may re-read the contents of the modified .ini file. I had not run the .exe after modifying the .ini file.

I replied back to him that the contents were changed and also the additional step was missed. These were the two reasons why I was able to reproduce the issue and he wasn't.

Learnings:

1. It is always safe to have a backup of files before and after conducting the test. Again it depends on what value this step adds to the overall mission. In my case, If I had not compared with the original file, there was very little chance that this issue would have been fixed before the customer found it.

2. Programmers have a natural tendency to fix issues on the fly. It is by nature that they fix issues and they do not consider it important to inform the tester about the changes. In my case, it turned out to be 100% true. The programmer never informed me about the change of text to 'Time' from 'Timer'.

3. I must have paid a bit more attention and come to the conclusion that the .exe must have been run to re-read from the .ini file. This taught me to relate the interaction between files & application to the tests I conduct using those files.

Simple things like having a backup of important files helped me find an important issue before it knocked the customer's door :)

Leia Mais…

Base state of a software application

This is a post on the importance of base state or initial state of a software application.
In the last post, I had mentioned about activating mobile office on my mobile.

I had not taken any note of the condition of my mobile connection settings. After I deactivated my mobile office, I was unable to access any websites on my mobile. After a lot of trial and error, setting up some other settings, I decided to approach the customer care and ask them to solve my problem. Finally I got it solved.

Observations:
> I was careless not to note the initial settings of the connection.
> I should have gathered more information about the new service(mobile office) as it did not support my access to wap enabled websites.

Connection with testing:
How many times have we come across a situation where we could not reproduce a defect or a behavior due to the change in the initial settings. We waste a lot of time reproducing the issue and finally realize that it was environment specific and had to ghost the system and start afresh. After this mobile experience, I realized the importance of noting the settings of my mobile. In testing, an environment of a test plays an equal role as the test itself.

Lessons learnt:
A tester has to know each of his test case/input. We never know which test may give the unwanted result/behavior.
How about running a test in our mind, imagining the probable consequences before running it in real. If testing was that simple, anyone could do testing.

Sometimes, we do need to probe a defect further to unearth even more critical defects. At the same time, we have to be careful to note the initial/base state of the application.

So, how do you deal with the ever changing state of a software?
Do you know each test you do?
How do you handle the issues you fail to reproduce at the first attempt?

Share your comments. And finally a question to you:
Is it necessary to know the initial/base state of the software under test?

Happy testing :)
-Ajay

Leia Mais…