Automated testing – the holy grail?

One of the most interesting debates that continues to rear it’s head is around automated testing, and people’s view as to what it is.

It ranges from seeing it as a panacea for all of the testing shortcomings, removing the need for testers as the automated tests do their jobs, through to automated tests being an optional extra if time permits.

I sit somewhere to the centre, not of the view that automated tests can replace people, but seeing the value that having a good set of repeatable automated tests can bring.

So, unpacking that a little, why do I think automated tests are a good thing?

  • Firstly, the concept of using a tool to help us to do our jobs easier has been around since our ancestors started to make flint tools for digging. We advance little without a tool of some sort, but we need to use the right tool, at the right time, for the right reason, otherwise it’s a waste of time.
  • Secondly, who wants to repeat the same tests manually over and over again? Been there, done that, and it’s not fun.
  • Thirdly, any time that you can save by manually repeating the same thing, means more time to think about and cover different test scenarios.
  • Fourthly, its an opportunity to learn a new skill – writing code to exercise code! For many people, it’s actually a fun challenge, but also gives us an insight into the mind of a developer. And that can only be an advantage.

The problems we have though is that some people (normally budget holders, normally outside of technology) believe that adopting automated tests means that they can dispense with testers. No! That is not the case. You cannot automate what you haven’t defined. Someone needs to use analytical thinking skills to determine what the test cases are, and to then test those cases.

The automated test tool is just that – a means to an end. There are skills that are needed to define the tests, and skills needed to write the code to make those tests run. Someone with a background in testing is still needed – whether you call them Quality Engineers, Developers in Test or Testers (I think another blog post on role names is pending!).

Used appropriately, automated tests complement other testing techniques, such as manual exploratory testing. There are different tools for functional UI testing, API testing, non-functional load and performance testing, so don’t be tempted to think that one tool is all that is needed. Selecting the right tool for the job is important.

There is a time element here – investment is needed in the right framework that is sustainable, and familiar to a number of team members, not just one person. It takes time to write automated tests, time to check failed tests, time to fix failed tests, time to maintain the framework and improve it as new ideas and techniques are discovered.

Automated tests cost time to create and maintain – but they save manual effort in execution. Used properly, automated tests can give confidence that:

  • A build has not broken fundamental use cases.
  • Functions built in previous sprints have not been broken (regression tests).
  • The performance of the application meets or exceeds expectations.
  • The application can handle the anticipated load.

I think it’s important to set some expectations here:

  • Automation itself is not the holy grail.
  • It is not a magic wand that gets rid of every problem.
  • It is also not an excuse to remove testing as a function from an organisation.
  • Automated tests cannot stop part way through and go off script, but a manual tester can decide to press the F5 or back button and see what happens. Exploratory testing!
  • Automated tests are a way of executing the same tests over and over again. That’s it!
  • Automated tests complement manual exploratory tests.
  • Automated tests are a tool to help the team.

Remember the saying ‘A fool with a tool is still a fool’.

So use the tools wisely.



Mind The Gap

Testing is all about the gaps.

Ask anyone what a software tester’s job is, and the answer will more than likely be along the lines of ‘to see if a bunch of code does what it is supposed to do or not’.

Ok, I’ve simplified it, but you get the idea. The perception is that it’s basically exercising software to see if it behaves as it should do, in accordance with the requirements.

But testing is more than that. Or it ought to be! It’s about finding the gaps between what a product owner/stakeholder asked for, and what was delivered – from a software behaviour perspective (at functional and non-functional levels) and also from a customer behaviour and usability perspective.

There may be gaps in the requirements that no-one has considered, gaps in the process that the end user will follow, or gaps in the understanding of the data flow between applications. A tester is in a great position to apply critical thinking skills to not just assess the requirements in terms of what is stated, but to identify what is missing, ask the awkward questions and feed that information back into the team.

I wonder though if we push this enough within the industry. So much emphasis is on the ‘automation’ side of things that we lose sight of the other areas where testers can (and should) add value. Automated tests are necessary and valuable, as we cannot manually regression test everything, and I am not saying that we should do so. But these are essentially ‘dumb’ tests, just repeating what they have been written to do. An individual with great critical thinking and analytical skills can be far more productive spending their time looking for gaps than writing automated tests, yet we seem to value writing automated tests over critical thinking. (See what I did there – good use of the Agile Manifesto syntax).

The writing of automated test scenarios is different to the actual automated tests themselves. The scenarios need to be thought out and established before they can be coded, and you need a particular skill to do that (what I term the testers mindset). The coding of the tests also requires a particular skill in developing code. Developers are great at writing code – it’s their bread and butter, so would it not make sense for testers to focus on the scenarios and ask the developers to code the tests?

I appreciate that there are testers who enjoy coding, so this wouldn’t suit everyone, but if I had to make a choice due to resource constraints, I would rather a tester focus on what needs to be tested, and let someone else code how it is done. After all – in order to automate a scenario, you must first define it.

I expect there will be many who disagree with me, and I welcome other opinions on this. As I state on the home page, this is just my ramblings based on my perception of the industry as it stands, and I could be mistaken, so if you know or feel differently, please do let me know.



Manual Testing is dead – long live Manual Testing!

This posting is a little later than planned (by about a week), as I had intended writing it after attending the National Software Testing Conference in London, where I was fortunate to speak as well as attend some great talks. I came away with 4 blog ideas, and this is the first one of them.

The demise of manual testing is being discussed in blogs, magazines, conferences, meetup’s etc. If you look at job ad’s, you’d think that manual testing has already died a death and been buried! They all mention ‘Automation tester’ – like its the ONLY thing that testers need to do. So, it was refreshing to attend sessions where people took a different view of things.

I’ve mentioned before about the need for testers to be able to do manual exploratory testing, and it was great to hear Ingo Philipp from Tricentis discuss this in a conference setting. As an industry we need to push testers back towards performing manual exploratory testing, to be complemented by automated regression testing, otherwise we are going to start missing defects due to deficiencies in the overall coverage.

Think about it. An automated test is only as good as the person who wrote it, and only as up to date as when it was last maintained. An automated test cannot make allowances for something that has changed. It cannot stop part-way through and think ‘I wonder what happens if I click this button rather than following the process flow’. It cannot look at the number of steps and highlight that the application sucks from a usability perspective. It cannot point out that the colour scheme is unreadable, or that the company logo is the wrong colour/shape/size etc. It can only run the steps it has been coded to do and validate against what is has been told to check for. So a test may pass, as the application displays what was expected, but what if additional text is present that shouldn’t be there? The test would pass, and unless anyone manually tested that screen, it would go undetected.

I’m not saying that automated tests are unimportant, far from it, but they have their place within a tester’s toolset and are not the only tool available to a tester. Of course you need to have automated tests in place to be able to follow a Continuous integration process, but automated tests cannot cover every possible scenario.

There is another aspect to this as well. Most applications that we develop are going to be used by human beings, so why do we insist on believing that it is best tested by code, without any human test covereage as well? Automated tests should cover the repetitive tests, the load and performance tests, but a person, a tester, needs to look at the application and think about the different paths that the end user could take.

The fun is in the thinking. The benefits are derived from the thinking. A tester’s brain is needed to assess the tests needed, and I despair when I read of really good manual testers with many years experience who feel that they have to leave the field of testing as they do not have automation experience. I sympathise as I came from a manual background. Coding holds no interest for me – if it did, I would have become a developer. So, my career has been based on testing software and working out how best to do so.

We need to stop this freefall ride into automation oblivion, and look at hiring and supporting multi-skilled testers. If a tester cannot write automated code, does it really matter? Better to have a tester who can look at a requirement and work out what needs testing, than a tester who can code but has no clue how to test something!
Developers can write code, so why not pair a develope with a tester to write the automated tests that a tester defines. If a tester wants to write code, and has the time to do so, then that’s great – but we should not be penalising people for knowing how to assess a requriement and define the tests needed (i.e. the core elements of the job), just because they do not have an additional skill in coding automation tests.

I do feel that there are a number of us trying to push back the tide a little to show people the benefits of doing both manual and automated testing, but the more that speakers such as Ingo and myself are getting out there and promoting the benefits of exploratory testing, the better. We need to stop this damaging trend, and ensure that we retain the best skilled testers before they feel undervalued and move on.

Now, who’s with me on this?


There are a number of things I have been involved with recently which have highlighted the importance and necessity of the Testing Community.

We are fortunate to work in an exciting industry – testing technologies, some ground-breaking and others less so, but what we do has an impact on others, and we like to think that our testing efforts result in better end products.

It’s this mindset which I think also feeds into some of the great collaboration that we see.

In my role, I co-run a Testers Chapter (we refer as the QA Chapter – I dislike using the term QA, but thats for another post). This was created in 2011, with no more than 20 testers from different groups within our organisation. As of May 2017, I have 2 people helping me to organise and run the sessions, and there are 85 invitees from the UK, USA and Europe. It’s staggering to be honest to look at the numbers and think that we have that many testers across the globe – many of whom have never met, and are unaware of each others existence. It is a fantastic achievement, and something I am proud to have created and fostered. Our internal community has helped many testers to share problems and solutions, ask for general guidance, and not to feel alone in their day jobs. Some of the testers work with others, but there are testers who are alone in a group and this really serves a need – people to reach out to and ask for help when needed. And that for me is why I do this.

Outside of work, there are many other communities of testers here in the UK, and I have started getting involved in mentoring, as part of the Ministry of Testing, and also the BCS Specialist Testing Interest group, as I want to give something back.

Actually there are a lot of other testers out there who are doing just that. I could name so many who have set up free meetups, training sessions etc (e.g. Rosie Sherry, Tony Bruce, Abby Bangser, Mark Winteringham, Dan Ashby, Richard Bradshaw to name a few), and what I love about this is how much time and effort people are prepared to give back – for nothing!

We are very lucky to be part of this, and I’d encourage you to get involved in your community – whether it is specific to a particular job discipline such as Testing, Business Analysis, or more general around Agile and delivery. It’s really rewarding to meet new people who you’d never get a chance to meet in your day job, and it’s fun to try out new things too.

If you haven’t connected with any other testers yet, why not do a quick Google search – within 5 minutes you’ll have found something going on near you.

Right, now to ‘walk the walk’ and book the next Testers Chapter….


Beating them at their own game!

This is a post about Google Chrome, the F12 Developer tool feature and how I felt very smug after using it to get around restrictions!!

I installed an Ad-blocker into Chrome, which is the main browser I used, and I noticed that certain sites were showing messages asking me to remove the Ad-blocker (like the images below), or sign-up for content – neither of which I want to do to be honest. I have ensured that the sites cannot be identified from the screenshots below, as it is not my intention to draw attention to any specific sites, as there are many out there that have these restrictions in place.


There was one particular article I wanted to read and I could see no reason why I couldnt do so, seeing it as it wasn’t something that was exclusive to this particular site. I could have searched elsewhere, but was feeling in a less than co-operative mood, so decided to have a play.

Pressing the F12 button and opening the Developer Tools gave me the chance to inspect the element on the page that was blocking the text, by clicking on the button (highlighted in yellow)….

….and then clicking on the blurred area on the page that I wanted to inspect:

This then showed the element in more detail and I could then investigate further.

I found that I had two different choices, depending upon the type of restrictions imposed:

  1. To read the plain text within the Developer Tools pane rather than on the screen, but that meant having to expand every element in order to reach each paragraph:
  2. To try removing the blocker itself in order to read the text on screen as intended by just deleting that line of text.

On one site I had to use option 1 to open each element to read it as deleting the element actually deleted the text within. On another site I used option 2, and simply deleted the element off the page and the text was then visible with no restrictions.

I was surprised how easy it was and I guess that over time, the website builders will try to make this more difficult to do, but not many people really know about the F12 function, so I feel it my duty to help spread the word a little.

It really is that simple. If you are not sure, just have a play. If you delete things that you didnt want to, just reload the page and try again. It really is satisfying to beat people at their own game sometimes!



The UKStar legacy


I’m back from the inaugural UKStar event in London, where I was privileged to co-present a talk with   on communication, and it seemed to go down really well. But that’s not the only reason that it was such an enjoyable time.

The venue – County Hall, with a view across Westminster Bridge to Parliament and Big Ben. As a Londoner I have never managed to see the view from there so it was something special. I do have to say that etc.venues did a good job of keeping us fed and watered, and the guys at Eurostar who organised the event were great. They all seemed to really enjoy the event and were super helpful.

The talks – so many great ones and I missed a couple due to clashes that I would love to have been in, but then thats always the case with dual tracks.
I attended the ‘Hey what just Hackened’ half day session with Declan O’Riordan, which helped us look at security testing in a new light, the keynote with Maaret Pyhäjärvi & Llewellyn Falco on the concept of Mob Testing was something new to me. I also enjoyed seeing Paul Collis of the FCA (who I know from the Test Management Forums) do his first conference talk about the transformation of the testing function, Dan Ashby & Hannah Mason speaking about the mentoring and learning opportunities at the Software Testing Clinic ( and also Stephen Janaway’s journey from Test Management to a broader Software Delivery Manager role gave me a lot of food for thought ( The talks were inspiring and I really like the fact that new speakers were encouraged.

The atmosphere – there was a noticable buzz for the whole two days, with so much interaction going on. I think this is one of the best things about any conference – a chance to meet old friends and make new ones, which I did during the Tuesday morning Lean Coffee session, ad-hoc chats over coffee and lunch, and a very nice evening meal after the Monday evening drinks.

So – would I encourage people to attend a conference? Yes! Choose carefully though, as there are many and you want to go to one with a good range of talks so you can maximise the opportunities to learn, contribute to discussions and network.

Would I encourage new speakers to apply? Absolutely! I’m really pleased to see a change in focus at events to actively encourage new speakers, and I will be helping at the BCS to mentor someone this year. I will also speak to Dan Ashby about opportunities to give something back to the testing community.

If you commit to do just one thing this year, then go to a great conference. Be challenged, learn new things and meet new people. What have you got to lose?

2017 – 3 weeks in.

A little belated, but I can get away with posting ‘Happy New Year‘ as this is my first post of 2017.

So, here we are, about 3 weeks in to the year and a lot has happened already:

  • I have taken on a new role in my company as Director of Quality Engineering – essentially Director of the QA function, for a different brand to where I worked before so this has opened up some new and exciting opportunities for me within
  • I recently joined the BCS and am hoping to get involved in their mentoring scheme for new conference speakers – I’d love to give something back to our testing community!
  • An article which I worked on late last year has been published in Test Magazine – and they have done a cracking job of the presentation and layout (thanks 31Media). Its all about where we should focus our efforts as testers when looking at honing our skills, so I would love to hear any feedback if you do happen to read it.
  • I am speaking at Testing Showcase North in Manchester in February on the subject of Tester Training, in a similar vein to the magazine article, but of course delivered as an interactive talk. Details available here
  • Then a week later I am joining my colleague Bhagya Perera in London at the inaugural UKStar event to deliver a session on ‘The Communication Bridge’, which I am very much looking forward to as well. More details here
  • Oh, and in May I will be speaking at the National Software Testing conference in London!

I cant believe how busy it’s been already, and that is without the workshop that I am running with my new team next Friday, and our ongoing QA Chapters that we run in-house.

But I am not complaining, I get bored easily (something that I am not proud of, as I wish I had more patience overall!), so doing a lot of things is good for me. One of my old team asked me last week how I found the time to do so many things. I do wonder myself sometimes, but my reply to her was that it comes down to having an in-built passion to do something of benefit. No-one can be forced to do anything extra – we have to want to. The secret is to find something that excites you, helps you grow as an individual, as well as in terms of job related skills. Giving something back by helping others where you can (it shouldn’t be just about personal gain), and making a difference – these are important to me, and I really hope that 2017 is even more awesome than 2016. Of course that depends on the amount of time and effort I am prepared to invest, so the incentive lies with me – but that’s what is good about it. I am in control and can do as much or as little as I feel capable of doing.

So, watch this space 🙂

Oh, one final thought – my job title is now Director, not Manager, but I think I will leave the site titled as ‘Musings of a Test Manager’, as I still think it sounds good to me. I tried thinking of alternatives, but ‘Doodles of a QA Director’ doesnt really have the same ring!