Friday, September 5, 2014

Software Testing, Some myths and the Path forward...

This blog post is about a topic close to my heart - Software Testing
It is going to be a long post highlighting few key points/myths and what I think about them.

Feel free to comment and we can discuss. Ok, let's start.

Case 1: Testing process
Scenario 1: 
You are supposed to test a feature. You receive a requirement document and there is a formal review process. You write test cases based on your understanding of the feature. Add a layer of review process. Start testing executing the test cases. File bugs. Measure how many test cases are executed per day per tester. Write a test case for every bug discovered, if they are found outside the test cases. Reject any new feature addition as it was not planned in your scope. You assure the quality of the product and tell when the product is ready for release.

Scenario 2:
You are suppose to test a feature. You figure out who is the decision maker, the stakeholders involved and try to understand your role. You seek help, remove traps to get access to the product, interact with the programmers and prepare a light-weight document (checklist/mindmap or on a simple sheet of paper ) which acts as a reference. Based on your interactions with people and products, you update your document. You test the product, file bugs, ask questions, seek help/tips from experts and inform the stakeholders about the status of the product and project.

I will let you pick the scenario you like.
If you are more familiar or in favor of Scenario 1, we will have lots to discuss.

Case 2: Manual Testing vs Automation Testing
Have you heard of the word 'Sapience'? Do you use just your hands while testing? Do you think? What happens in your brain when you test? Think for a minute. Yes, THINK.
I feel that the very notion of classifying testing as manual vs automated is wrong. Michael and James have come up with an excellent blog post here highlighting the difference between testing and checking.
Here is the diagram from their post.


 Also, check out this blog post by Michael - http://www.developsense.com/blog/2013/02/manual-and-automated-testing/

These two posts talk about Testing and Checking:
 http://www.satisfice.com/blog/archives/358
 http://www.satisfice.com/blog/archives/856

What is testing then?
Before reading more on what testing is, do check out the slides from BBST Foundations which describe what a computer program is. The same folks who would define a computer program as a set of instructions would define testing as a process to find bugs or assure quality of the product and testers as the gatekeepers of quality.
 


Check out an excellent video on the talk between devil and angel of software testing:

Testing is an intellectual activity. If you can think well, you can test well. Every test is a question to the product, sometimes to the project stakeholder. Check out the lessons #16 and #19

Testing is not about test cases or documentation or tools. These were designed to help you test better, support testing. Instead, what do we see testers focusing on?

Learning new tools, writing better user stories, automating tasks and dreaming of days when testing would be fully automated!!!

Pick any resume and check the Skills section: 7 out of 10 would mention some or the other tool names.
Quick Learning is a skill. Tool is a tool. A fool with a tool is still a fool (maybe, a dangerous fool).
So, what am I proposing?
Focus on skills like Observation, Thinking (Critical, Lateral, Forward/Backward, Parallel, Technical), Reasoning, Preparation, Programming basics, Collaboration, Framing, Reporting, Bug Hunting, Social Skills, Knowledge about Psychology, understand and apply oracles and heuristics, quick learning and many more skills. I suggest two books - Lessons Learned in Software Testing (Cem Kaner, James Bach, Bret Pettichord) and Testing Computer Software (Cem Kaner, Jack Falk, Hung Q Nguyen)

Practice your testing skills at Weekend Testing
Read the blogs everyday from this feed - Ministry of Testing
Watch the videos from BBST courses - Videos
Build your test ideas repository -
Test Heuristics Cheat Sheet,
Mnemonics,
YANDY list
300 common software errors
Pick your topic of interest and become a specialist in that. The Ministry of Testing resources page can help you get started. Resources

Case 3: Automation is the future
If you have been hearing of new tools, new languages every week and wondering which one to learn, here is a simple tip:
Your job is safe if you have the skills. There is no single tool which can replace a working human brain. If tools can replace humans, we would have been jobless long back. If you think automation is the future, I am sorry. You are missing a very key element here - EMOTIONS. Check out the ppt file here - Emotions And Test Oracles by Michael Bolton.

Google for TDD and check how many links highlight why it does not work.
If you think that testers need to learn coding, I am not against it completely. My only question is: How often have we asked programmers to learn testing?

If you think that TDD is the future, remind yourself again that they are good for programmers and do not replace/negate the role of a tester. Read the first comment on this post: http://www.satisfice.com/blog/archives/638

Here is the bumper post: FDA highlighting exploratory approach. 
And if you still think that tools can test mobile applications and there will be no need for testers, I will assume that you have no idea about this excellent keynote by Jonathan Kohl.

And I leave you with this final post: Testers: Get out of Quality Assurance Business by Michael Bolton. 

Leia Mais…

Saturday, July 26, 2014

Software Testing World Cup and our experience!!!

Hope most of you are aware of the awesome software testing competition conducted by Matt Heusser, Maik Nogens and many volunteers across the world. The whole concept of competing for one spot per continent is awesome. You might have participated in contests within your company or a local gathering but can it get bigger than a continent? Software Testing World Cup lets you experience that moment where you compete with close to 250 teams. Yes, 250 and only 1 winner.

Before I begin highlighting our team's experience, I want to thank the lead organizers - Matt Heusser and Maik Nogens and all the volunteer judges, the sponsors and everyone who helped conduct this BIG contest at the highest level. Check out the website for more details: Software Testing World Cup website

The logo is quite good too :)
For the Asia preliminaries, we had started preparing from April. We were a team of four - Pranav, Satish, Sundar and myself. I created a folder on Google Drive and shared it with other teams representing Fiberlink, an IBM company. Two teams were representing us from the America continent. We had few documents, list of test websites, tutorials on the G Drive. We also scheduled for 30 minute testing sessions till 07th June. Due to our busy schedule, we could manage only 6-7 focused sessions. We tested websites, mobile apps and windows applications too.

Meanwhile, we were brushing up our knowledge on different bugs, patterns, quick tests and checklists. We not only practiced the testing sessions but we also practiced writing the final test reports. We did send a sample test report to few judges for feedback and this exercise helped.

On the day of contest - 07th July, we were all set to test and only after an hour of testing, we realized that many other teams were not able to access the SUT and the contest was called off.
We were worried about the future date as one of us had to fly and we were not sure on how we would manage testing across time zones. The date was announced - 25th July and we had the following conflicting events:
- The day after a hectic office trip where I enjoyed a lot and was tired by end of day
- Pranav had to be part of interview drive for developers
- Protest in Bangalore
- Sundar had to pack for his trip that night

We did not have any practice sessions after the contest was postponed. We were monitoring the other continents' contests closely. We planned to assemble at 8.30 am IST. As Pranav was busy with the drive, we asked Rahul to join us. One interesting observation from my experience is that people find it tough to clear traps till someone else clears similar trap. I reached at 8.30 am and logged in to my Agile Test manager account only to find that the password was incorrect. On attempting Forgot password, it asked for Security Answers which I had no clue of. Immediately, I pinged Michael from HP, Maik and Smita asking them to help me with my login.

Michael solved it immediately and I was able to login to discover that it displayed Europe instance instead of Asia. After confirming that it was an issue on their side, I was waiting for the organizers to announce the SUT. Sundar and Rahul joined me in the meeting room and Satish was stuck in traffic because of the protest.
Sundar tried connecting to mIRC on both the laptops but it kept retrying and connection was timed out. Then, Sundar did something brilliant - connected to VPN and tried connecting via US network and we were placed in the chat room immediately :) Many teams from our company backed out as they were not able to connect to mIRC chat.

I use Unroll.me and found it great to start with. The only disadvantage I face with it is few important emails - like the one from STWC announcing the SUT get moved directly to the Unroll folder. We were informed a day before the contest that there will be two websites and one windows application to choose from. I was asking Sundar on which application to choose - Website or Windows application. He replied NBA.com and I asked him why NBA without realizing that the SUT was already announced.

Immediately, we started our task and we listed out our focus areas for the next 30 minutes. Satish was testing from the cab and sharing files over Skype. We had the Skype group chat going where each one would highlight the bug and send the screenshot. My role was to replicate, file the bug and do some testing when the bug flow was manageable. Sundar was focusing on the different tours, I took up the quick tests with different checkers and Rahul was testing on iPad. Satish was testing on Android smartphone.

We found quite a decent number of critical bugs and in no time, there was just 45 minutes left.
We covered most of the areas planned for testing and I started consolidating the report. Sundar was filing the bugs discovered by Satish and Rahul. He was also helping me with the mind map for additional testing scenarios. At 12.31pm, we were done with testing and finalizing the test report. We emailed it to the mentioned email address and quickly received a confirmation from organizers that they received the report.

Key Highlights:
- Previous experience of working under severe time pressure helps.
- Getting our test reports reviewed by the judges pointed us to key areas we had overlooked.
- Practice sessions helped us finalize on a convention within the team.
- Do not panic when you face a trouble - think through and take steps to solve it.

That was one contest well planned and executed :)
Now, waiting for the results.









Leia Mais…