Nicky Tests Software

Thursday, October 12, 2017

Introducing people to Exploratory Testing Part I

 A bit of context

For the past 2 months(ish) I've been working on introducing Exploratory Testing to people in my project, starting with people in my immediate team of 3 testers, which is distributed between 3 countries. The project, as a whole, has a lot more than that, but the plan is to introduce  this as a pilot, see what the testers think of it, and then (hopefully) introduce this approach to other teams and other features.

I'm still a relatively new person on this project as I've been on it for 6 months, but the other testers in my team have been on this project for 3-5 years. So I've made sure to ask their thoughts, listen to their ideas and address their concerns about this - they know things about our context (which I don't) because of their experience here.

Currently, on the project, we write test scripts, link them to test cases, then execute those test cases. I'm under the impression that a lot of people on the project have only ever used test cases to formally do testing (when they're not doing "Exploratory Testing").

Another thing to keep in mind, is that this process is still ongoing (hence 'Part I'), but while these thoughts are still fresh on my mind, I wanted to get them down.

Sites/resources I shared

Spotify Offline: Exploratory Testing by Rikard Edgren
I asked my Test Manager (who also thinks Exploratory Testing is effective) about resources I can share and she recommended two Youtube videos by Rikard Edgren.

James Bach has a lot of useful posts on his blog helping explain what Exploratory Testing is.
Few examples:
Exploratory Testing 3.0
What is Exploratory Testing

I also shared a post from Michael Bolton's series - what Exploratory Testing is not

Lastly, I decided that Session Based Test Management (SBTM) would be a great way to help us structure our Exploratory Testing, so I shared some resources around that including this powerpoint by Anders Claesson

Managing others' expectations

I've noticed that a lot of people on this project have a very different understanding (to me) of what Exploratory Testing is. Based on what people on this project say, it seems that they think Exploratory Testing and Ad hoc testing are the same thing.

Since initially introducing the idea to the other two testers I work with, I'd say I've been met with cautious reception. Test cases is seen, by one, as proper testing and Exploratory Testing is not-  I'm still working on breaking that misconception. Aside from that, it does seem to be a welcome idea - you get to see results faster and are able to react to what you find as you test.

In terms of time estimates and how this affects our team meeting it's goals - I've been sure to communicate with our team that this is a new way of working which we need to learn - so any time savings may not be seen straight away.

Lastly, there is the idea of coverage - to cover this, I've decided to specifically mention, at the start of each charter, which Acceptance Criteria is covered in the charter. People on this project like reports and seeing the number of passed test cases etc. - I'm still learning how to deal with that mindset and any obstacles which arise there.

Managing my own expectations

This has been tough. It's been a while since I've been in a work environment with people avid fans of test cases. I don't think that test cases offer no value at all, but I think people overestimate the value it provides.  Test cases can give people a false sense of security of the state of the product when they see that 95% of their test cases have passed, but they can't properly attach meaning to how a 95% pass rate affects the customer. It's just a number.

I've also been trying to manage my expectations around other people's understanding of good testing and my own understanding of good testing. I constantly remind myself that their understanding is based on their own experiences. So neither of us are necessarily wrong - we are both right in our own mind.

My goal is to effectively show people on my project another way of doing testing. Then they can make more informed decisions in the future and choose which approach is best for their context.

Moving Forward

I'm hoping to sit with the tester who'll soon be on-site and pair test with them as we do Exploratory Testing. I should also organise a pair testing session with the tester who'll still be offsite. Once we've done this, I'll seek more feedback on this approach and see what they like and what they are concerned about (in the context of this project). We're also working on a Low Tech Dashboard to help us communicate the testing status for our features and help others attach meaning to what we present. 

In time, I'm hoping to help introduce this approach to other teams in our project - but for now we need to continue the pilot first. 

Tuesday, May 30, 2017

The limitations of Acceptance Criteria

According to Software Testing Class, Acceptance Criteria are conditions which a software application should satisfy to be accepted by a user or customer.

Often these can also be used to guide the testing for a testing team. If the acceptance criteria are met, then the story has passed. You can choose to test strictly against the acceptance criteria by using test cases or exploratory testing etc. and then once each acceptance criteria has been "ticked off", you can mark testing as done.

The thing is - acceptance criteria has its limitations.

You are expecting someone to know in advance, before seeing the software, exactly how the software application should be. So if you are testing strictly against the Acceptance Criteria - you are in essence trusting that, that person (or group of people) who wrote the acceptance criteria knows everything about what is needed before the software is built.

People don't know what they want until they see it (same goes for knowing what they don't want)
I think it's possible to build a product that the customer did say they want then still find them to be unhappy because they realised once they saw it - that it wasn't quite what they wanted. After seeing something they are then better able to articulate what they want or need from the software application. They may not, however, be able to articulate what they want clearly - until they have something in front of them.

Here's a fun analogy to help me explain this further
Finding a romantic partner
Now of course finding a romantic partner and acceptance criteria are vastly different things, but let me explain. If you've ever, as a single person, talked to your friends etc. about what you want in a potential romantic partner, you may rattle off things such as:

  • Funny
  • Kind
  • Attractive
  • Likes sports
  • Same religion

Among other things, and these are things you may deem to be important. So let's assume the above criteria are all must-haves, they are your "acceptance criteria".

But as I said before - acceptance criteria has its limitations.

If you think you know exactly what you want when it comes to a romantic partner, then technically a friend can introduce you to someone who is Funny, Kind etc. and you'll be happy.

But it's not that simple, while there are some things that you may not want to compromise on, there are also some things you may not realise are important to you, until you have met someone who has those special qualities, which you didn't think to define. (or they may be missing some qualities that you didn't think to define)

 Image courtesy of

Wednesday, May 24, 2017

My Testbash Belfast 2017 Experience Report Part I

This is a two-part Experience Report, the first part will cover preparing my talk, the pre-Testbash Shindig and the first half of the conference. The second part will cover the second half of the conference and the Post-Testbash shindig.

Preparing my talk

I started preparing my talk around 2-2.5 months before the conference. But I didn't properly gain momentum until about 1.5 months before my talk. Initially I tried to write the whole talk in Google Docs - but I found that didn't work for me. Instead, I ended up creating the slides and writing speaker notes below.

I aimed to have a completed presentation ASAP and then just edit it continuously up until I gave my talk. I find it a lot easier to edit a presentation that's complete than to add more to one that is incomplete.

Tuesday, May 16, 2017

My Experience at Romanian Testing Conference 2017 - told through photos

I thought I'd share my experience at Romanian Testing Conference 2017 with the use of photos :)

Here are some photos of our talented speakers on the evening of Thursday 11 May, before the main conference day.

Here's Rob Lambert , the conference chair, welcoming all of us to the Romanian Testing Conference 2017

One of my favourite slides from Santhosh Tuppad's opening keynote

Some photos from Adam Knight's talk on communicating risk

Marcel Gehlen sharing his expertise on creating a test friendly environment

One of the slides from Elizabeth Zagroba's talk on how to succeed as an introvert

At lunch we had quite the dessert offering, I ate less than half of what's on this plate. It was very rich. But I saw lots of other people eat twice the amount on this plate for dessert. 

My certificate :)

Harry Girlea giving his closing keynote

Sightseeing - I'm doing my classic thumbs up pose here

The Opera House in Cluj

View over Cluj

Going for a walk in central park

Monday, May 15, 2017

Don't call it exploratory testing (if it's not exploratory testing)

Well, this is a bit of a rant - but seriously.

I don't like people calling an activity exploratory testing when it's actually ad hoc testing.

Get it right.

I think using the term "exploratory testing" loosely - takes away from the value that actual exploratory testing can add to a project.

Below are a few questions to ask yourself, to see if you are actually doing exploratory testing (or if it's in fact ad hoc testing)

  • Are you doing concurrent test design and test execution at the same time?
  • Are past findings influencing what you do next? 
  • Have you written some test ideas or goals in mind - to help you explore the application?
  • Are you documenting your test session?
  • Are you focussing on both positive and negative scenarios? (if you're just looking for bugs - this is likely to be ad hoc testing)
  • Can you explain the process/what you did during the test session to someone else?
  • Are you a skilled tester? (ad hoc can be done by anyone, doing proper exploratory testing is a skill in itself)
  • Are you doing SBTM (Session Based Test Management)? This is a way to structure your exploratory testing. If you are doing SBTM, this is a good way to indicate you are doing Exploratory Testing. BUT, if you're not doing SBTM, that doesn't necessarily mean you're not doing Exploratory Testing.

Updated: I  have updated this blog post to help explain the difference between ad hoc testing and exploratory testing by posing a few questions

Monday, May 8, 2017

Interview with Mark Winteringham

In his own words, here's a bit about Mark Winteringham:

I am a tester, coach, mentor, teacher and international speaker, presenting workshops and talks on technical testing techniques. I’ve worked on award winning projects across a wide variety of technology sectors ranging from broadcast, digital, financial and public sector working with various Web, mobile and desktop technologies.

I’m an expert in technical testing and test automation and a passionate advocate of risk-based automation and automation in testing practices which I regularly blog about at and the co-founder of the Software Testing Clinic. in London, a regular workshop for new and junior testers to receive free mentoring and lessons in software testing. I also have a keen interest in various technologies, developing new apps and Internet of thing devices regularly. You can get in touch with me on Twitter: @2bittester

I noticed that the next event for Software Testing Clinic is sold out for both mentors and students - any plans to expand in the near future?

Wednesday, May 3, 2017

Getting started on a testing project

I started on a new project a few weeks ago and thought it would be a good idea to share a checklist for what new testers on a project need and some good starting questions when you, as a tester, are new on a project

Checklist for what new testers on a project need (Note, your project may not include all of the below)

Note to check if user credentials are needed for any of the below