Category Archives: Test Automation

Test Automation related content

QA – Creating a high performing QA function

Posted on by
Reply

Two years back I was asked to put together a presentation on how to create a high performing QA function for a large financial company who were considering an Agile transformation.

The presentation touches upon all aspects of Agile; process and practices. It covers continuous integration, continuous delivery, test automation, Scrum process, defect management, etc…

Though the presentation below does contain some specifics to the actual company, it does have a lot of reusable content that you may find useful when evaluating your own company QA function needs within your Agile implementation.



Recently I had the pleasure of working with Gareth Waterhouse, he has an indepth knowledge of testing within an Agile environment as well as a very strong knowledge of test automation and I would highly recommend visiting his blog for everything about QA within an Agile development team.

Continuous Delivery – Its more than just CD ;-)

Posted on by
Reply

CD (note I’ve deliberately used its abbreviation) is cool… but don’t despair if your not able to delivery the text book definition of CD, there are a number of flavors of CD;

  • Continuous Delivery
  • Continuous Deploy
  • Continuous Deploy and Test
  • Automated Continuous Delivery
  • Automated Continuous Deploy and Test

Some may ask why I’ve missed off Continuous Delivery and Test, you may choose to deploy your package to a test environment, but to deliver something that is not tested would be silly. I like to distinguish between delivery and deploy. Delivery is something that is ready to be delivered, ideally assembled (like a solid wood dining table) and that deploy is an express of something being sent that is not complete in some kind of form (flat packed dining room table from Ikea).

Continuous delivery is often used incorrectly out of context; I have heard many people use it to describe the whole pipeline, Continuous delivery is the smallest and least complex part of delivery; Continuous Integration, Test Automation, Configuration Management and Build Management are far more difficult to achieve and Delivery cannot be achieved at least without ‘Integration’, ‘Configuration’ and ‘Build’ (Test should be there, but technically you could deliver an unfinished product, its unlikely, but you could).

I’m currently involved in a project that aims to role out CD (flavor TBD, but highly like Automated Continuous Deploy and Test) on a enterprise legacy suite written in a language which is barely supported by any third party tools :-) .

For those asking, ‘but Carl your an Agile Expert, how does this fit within Agile…’, Continuous Delivery is a Agile’ish technical practice (could be ran in a waterfall model, depending on your setup), but you will gain huge benefits to your working processes by mitigating potential choke points and reducing lead times (test automation and software configuration in particular) whilst increasing your QA.

There are a number of areas you will need to take into consideration when planning your CD pipeline;

  • Test automation
  • Configuration management
  • Continuous Integration
  • Build management
  • Deploy / Delivery

A word to the wise, don’t get caught up in all of the hype surrounding CD. There’s a great book out about Continuous Delivery, but treat this with a measured approach. For example the 20min from commit to delivery may not be feasible for your business, it may take 20hrs, but judge if trying to reduce that lead time has value to your product instead of blanket aiming for the 20min barrier.

A great example is that some of my previous employers customers would only update once every five years, is it worth spending hundreds of thousands of dollars on reducing the lead time for a build down from 2 hrs to 20mins? As with everything Agile, find the sweet spot, don’t over bake it, but instead get it ‘just right’.

Continuous delivery, test automation and UA(T)

Posted on by
Reply

Everyone talks about UAT, but having a UAT phase is in many ways failure of your development model. Why should a user have to run testing if the requirements were captured correctly through continued engagement throughout the development (using sprint showcase)? Why does a user need to test the software when your automated test coverage should be 100% (unit, functional, GUI, performance, etc..)? Are you building total product or developing against features? Spending hours / days on devising a UAT checklist or strategy?

I have put the (T) of UAT in brackets as part of your continuous delivery model should incorporate the concept of omitting the testing element and instead the user should just be signing off against a deployment to a staging environment.

User Acceptance is an improved model on User Acceptance Testing, it’ll minimise the need for User Acceptance and stop you from having to put together a user testing strategy.

 

Over the GUI test automation, Excel and a little VBA

Posted on by
Reply

Automated testing via the GUI is a risky business and should be avoided as a testing method (Winrunner, Sikuli, EggPlant, etc…). Most of these tools use Bitmap comparisons to identify the area’s that the script should work in, the problem with Bitmap comparison tools are that they can easily fail if the UI changes (just a few pixels can often confuse bitmap comparisons), even when the UI doesn’t change repetitive layouts (cell’s in a spreadsheet) pose a problem for Bitmap comparison tools.

We often explain to teams that they should use the layer below the GUI (GUI’s should be a thin layer on the stack), a command line interface, macro mode, API, etc…, but alot of people struggle with this concept.

Think about using Excel, as a tester I need to automate setting different cell values as part of my test case. I could use an over the GUI tool to simulate the user clicking between cells and set the values, but more often than not a GUI testing tool isn’t going to be able to distinguish between one cell and another (give it a go, all cells look the same :-) ), therefore causing this test to fail.

Below is some very basic VBA setting values in a Spreadsheet;

Sub SetValues()
Range(“A1″).Value = “Test Value Cell A1″
Range(“P2″).Value = “Test Value Cell P2″
Range(“R9″).Value = “Test Value Cell R9″
End Sub

This code will set the values of the cell’s programatically, therefore removing the need to set the cell values ‘over’ the GUI and also removing the risk of not setting the correct cells with the correct values.

Over the GUI testing often appears reasonably precise to run with when starting out, but once automated and scaled up (particular if part of your continuous delivery pipeline) you will find it fails more often than not and becomes a QA nightmare.