The first significant decision was to architect the regression solution in three distinct parts each with a clear set of responsibilities.

1. The framework projects accommodated the drivers Selenium and Appium, web and mobile respectively. Waiting to obey commands, these sat at the top of the stack and were responsible for interaction with the applications in ‘outside world’. Designed such that should we want to move from the currently available drivers we could without any impact on the rest of the framework.

2. The page models were the brains of the operation, the ‘guys’ with the intimate knowledge of the application. Each page model mirrored a corresponding rendered page or screen in the application and knew exactly how to reference the UI elements in order to control and manipulate them through the drivers.

3. The regression test projects, one project per application. BDD influenced and scenario driven, the deliberate use of generic declarative terminology helped to avoid the tests becoming too ‘brittle’ and changes to an evolving application and it’s implementation could be absorbed by the page models, minimising the need to refactor the tests. The assertions could remain unchanged because the behaviour and functionality of the application has not changed.

These high level design goals emerged and were refined through walkthroughs and peer reviews with the team, the importance and value of collaboration with talented people cannot be overstated.

With the building blocks in place the first tentative tests went in to ‘kick-the-tyres’ and prove the approach the canonical user login scenarios ( I wish I had a pound coin for every time I had seen that one ) but we had to start somewhere…. and it went in well without too much ‘complaint’… in fact, it just felt ‘right’, something that is hard to quantify but there was an overwhelming sense of everything had it’s place and everyone knows their duty… a bit like MVC when you know the convention you instinctively know where to put things… a wonderful feeling.

A significant unexpected discovery that was made on our journey was a SaaS BDD tool called HipTest. It was discovered by our QA team as they scoured the landscape for something that would allow them to capture features and supported Gherkin based scenarios, but also seamlessly integrated into Jira. HipTest was a real find and was the ‘icing on the cake’ in our end-to-end workflow. HipTest in a nutshell allows the flexible and fluent creation of features and scenarios, with rich, deep support for Gherkin through action words, which can then be exported to one of many target consuming technologies. In our case Specflow, with a single click our .feature files are generated ready to be consumed in the regression framework. Plus it integrates beautifully with Jira in both directions merging seamlessly the business requirements tooling and the development project tooling.

 

It is interesting to note that our native mobile ‘Point Of Care’ Android app was introduced after the initial framework had been modelled around Selenium Web driver, but the fundamental design, architecture and core concepts hold true making the implementation of Appium a straightforward process.

We now have three applications in the regression solution and are preparing for the wholesale migration of more manual test cases into HipTest to be organised into features and scenarios, which will then drive the next phase of the roadmap towards fully automated regression testing. Our mobile app has almost 300 automated tests providing a high level of confidence that the application is behaving correctly.

All new development now incorporates regression tests in the definition of done so moving forward we are adding the regression tests as we go.

Something that has emerged as a essential ingredient to the success is that the initial tests were carefully crafted and refined such that an exemplary implementation has been standardised for others to follow, it has been seen that the framework is relatively easy to pick up and extend by developers from different teams, especially our off-shore partners who are using the framework with minimal training, a testament to the adopted conventions and intuitive design.

Another aspect of the project worthy of a mention is reporting. NUnit generates output files as it executes and captures the results of the tests. Specflow tools have the ability to consume this output and generate reports that provide both an overview summary of the test run but also detailed information of each test case. Another endorsement of the decision to use BDD and scenarios is that the test reports are human-readable and so we are to evidence to the business stakeholders the levels of test coverage at the feature level using terminology that they understand.

Our regression framework is now fully integrated into our continuous delivery pipelines and we have harnessed the power of the cloud to develop a bespoke ‘test-cloud’ to run our Appium mobile tests on Android avd emulators based on different device configurations which is promising.

We still have a long way to go but the future is looking good for our regression framework as we become increasing confident in it’s stability and capabilities, and excitement grows that with each cautious step we are edging nearer to our goal. A goal that will see our QA team free from the mundane cycles of regression testing and liberated to apply their extensive product and system knowledge to do the essential exploratory testing that is often compromised. A goal that will see iCareHealth delivering “quality software at pace” that adds real value to the daily lives of our delighted customers.