Introducing ATS3

I was talking last time about an automated testing architecture with one “frontend” and many “backends”. We develop the frontend, which accepts a package drop, extracts the package testing info from it, and despatches it to the backend that this information nominates to do the testing. This backend will be an automated testing solution that somebody has contributed to Symbian.

This plan means that if you want to contribute code to Symbian, and you’ve got a testing solution for it that we don’t have, you have the choice of contributing your testing solution as well, and we’ll consider taking it and supporting it on its merits.

As of now, only one automated testing solution has been contributed. That is Nokia’s ATS3 (Automated Test System V3). In fact it hasn’t even really been contributed yet. We’re getting regular drops of the binary installation, but the code is still not open-source. That will come with ATS4, sometime later this year or early next.

Over the last couple of months I’ve been bringing up ATS3 in Symbian to give us an initial automated test capability. It’s a powerful, flexible and scaleable solution and,  by the standards of in-house enterprise development, it’s well done. It’s very responsively supported by Nokia’s ATS development team and they ship a release every 2 – 4 weeks.

ATS is automation at a higher level than specific automated testing applications. It’s really an automated test management system. In ATS terminology, a specific automated testing application is a harness. ATS can automate tests for all of the harnesses that Nokia themselves use, and it also supports a “generic” harness, that can wrap any automated testing application that’s capable of being controlled via a fairly adaptable XML protocol. So harnesses are “pluggable” for ATS, though the plugging is pretty fiddly.

ATS has a client-server-worker architecture. An ATS server accepts test-requests from clients and despatches them to ATS workers for execution. When you install the ATS worker component on a machine, you tell it the DNS name of the server you want it to work for. Then whenever that worker starts up, it will register its availability with that server. This makes load distribution and scaling a piece of cake.

The server’s software stack consists of Java, MySQL, Apache Tomcat and STAF – IBM’s open source System Testing Automation Framework. Both STAF and Tomcat depend on Java. The server controls a MySQL database in which all persistent data is stored. A Tomcat web-app provides a browser based GUI for the server.  STAF supports all the server-worker communications.

The worker’s software stack is just Java and STAF. On the client side, you just need your browser and some simple shell scripts for posting test requests to the server.

An ATS worker hosts 1 or more devices. A device is just a set of hardware and/or software resources that support the execution of tests for some harness. So a device could be a a phone, or a hardware reference board, or an installation of a Symbian OS emulator.

The ATS server needs to know about all the devices at its disposal. You define a device by writing a properties file. You register the device with the server by uploading this properties file through the server GUI and binding the device it to the worker machine that hosts it. Then device becomes available whenever that worker is available.

You can request a test run by logging into the server’s Tomcat GUI and manually entering the details. But automation is more to the point. Client-side software can request a test run by posting an action URL to the server web-app. There’s no special client needed to do this. You can do it cURL or anything that’s able to post a URL to an http host.

A test request posted via URL will tell the server the LAN location of a testdrop that specifies the test run you want. A testdrop is a zip package that contains (a) an XML recipe for prepping and executing the test with a specified harness, (b) a manfest of any files that have to be deployed on the device for the test prep, (c) all the files mentioned in the manifest, if any.  The server will then retrieve the testdrop, read it, and despatch a test run accordingly.

Optionally, the XML recipe can name a specific device on which the test must be run. In that case the server will queue the test for execution on that device when it becomes available. Otherwise, the server will look at the harness the test drop specifies and queue the test for execution on the first available device that supports this harness. You can also schedule tests for execution at times of your choosing, but I’ll skip that. A test run produces and test log and a report of the test results. These are retained by the server until you delete them. You can browse them or download them.

Altogether there’s a good case for making ATS itself the “frontend” for Symbian’s whole automated testing solution. But I don’t think it;s quite good enough. Getting ATS to run even the simplest test for you, even on an emulator, is a fiddly thing to set up – and you really have to be a inside user on the LAN that hosts the ATS network:  I’d like something with simpler needs at least for basic use, and I’d rather our frontend service relied only on the internet, so it could be opened up to the contributor community at large.

This is not to say however that ATS could not be used to do the heavy lifting of automated test management for us, if we develop “simplification layer” to go around it and make that layer live on internet protocols. This idea appeals, because ATS can do everything I think our frontend automation calls for, and we won’t have to maintain it.