Community-driven software QA. My quest for inspiration.
Scene-setter for blogs to come.
I haven’t mentioned since blog #1 that I’m the Test Lead at Symbian. Nine months into the job, it’s sunk in that what it must mainly be about is enabling. Enabling the Symbian community to forge the testing capability it needs.
Probably I’ve been slow to digest this insight because of my prior CV. Until I started this job, for going on 5 years I’d led the team that maintained the Developer Build & Test System (DABS) in Symbian’s corporate precursor, Symbian Software Ltd (call it SSL). DABS was the nightly build and test system shared by the 30 or so software development teams that made up the software engineering force in SSL. In SSL, naturally, it went without saying that we test our software to shippable quality, all by ourselves before anybody else gets a look at it. And DABS bore most of that testing effort.
Now, I can see that I my professional mindset needs reset. So I’ve been looking at Linux distros to see what I – and Symbian – can learn about successful test strategies for a non-profit provider of an open-source personal computing platform; foremost, to learn how these projects are able to harness community resources for their testing operations and the ways in which they do it.
I looked to Linux distributions because they are only sustained examples of successful non-profit providers of open-source personal computing platforms and are exemplary of community powered software production at its most accomplished and ambitious. Symbian by its raison d’être faces the same challenges that face a Linux distro. (But not only the same challenges; a qualification that might be crucial).
I use “non-profit” there in a relaxed sense. I mean that the platform is provided at no charge. All of the most popular Linux distros now – say the distrowatch top 5 – have some sort of corporate backing that turns a profit from professional services and support for the free platform, or by selling and supporting a value-added down-stream spin. Symbian has profit-motivated corporate backing too – our funding members – but our platform is provided at no charge, and nothing is success for Symbian but community-built success.
I’ve studied the web-presenses of leading Linux distros – Debian, Fedora, Mandriva, openSUSE, Ubuntu (impartially alphabetical order here). I’ve peered around the online resources and activities that lie behind. I’ve made some contacts who are in roles like mine, and been welcomed to question what they do, and how and why. (Debian, I must hasten to emphasise, has no corporate stakeholders: it is a pure community project).
I’ve taken away a rich load of perceptions and learning points about community-driven software testing and QA for which I want to get the attention of Symbian, its community and ecosystem. I’ll be blogging on this theme. Not all of my perceptions are of beckoning opportunities for Symbian to catch up. Some of them are of daunting impediments for Symbian catching up.
I’ve also taken away a sense of obligation to reciprocate the openess of the projects I’ve researched. I’m thinking: In the light of everything I’ve learned, what do I know about testing and QA that could be useful to these and other open-source projects? I’ll blog on that theme too.
Back to Syborg on Linux, with GCCE
In my “Syborg” article I made the point that Syborg – the Symbian kernel port that runs on QEMU – is a breakthrough for Symbian development on Linux. Symbian virtual devices with the Syborg kernel, running in the QEMU virtual machine environment, offer the same emulated Symbian development target under both Windows and Linux. The Symbian Emulator of yore was Windows-bound.
I did some experimental work to show that, with fairly superficial hacks, the Syborg kernel could be built, and a Symbian textshell ROM could be built and run on Ubuntu Linux using the code base of Symbian^2.0.0. I wrote up my findings and posted them on the Symbian developer wiki:
How to build the Syborg/QEMU baseport for Symbian^2.0.0 on Ubuntu Linux
How to build Symbian QEMU on Ubuntu Linux
How to build and run a Syborg/QEMU textshell ROM on Ubuntu Linux
Toward Symbian development by open source means
These efforts represent initial moves toward the goal of making Symbian development doable with wholly open source means.
That goal entails making Symbian development doable on Linux. So knowing that it’s already practical to build and run a rudimentary Symbian vm on Linux is an important encouragement.
Those efforts fell short, however, of demonstrating how to build and run a Symbian vm by wholly open source means. I cheated on three points:
- I didn’t have an open source compiler that could build Syborg. I used ARM’s proprietory RVCT compiler v2.2, for which Symbian has licenses.
- I didn’t build the necessary ROM-building tools for Linux. So far, Symbian kits provide only Windows executables for these tools and though they are open source, they haven’t been ported to Linux. I let Ubuntu execute the Windows executables via the Wine program loader.
- I didn’t have an open source installation of the Symbian build system, sbsv2 (a.k.a. “raptor”). Raptor is also open source, and runs on Linux, but the Linux installer for Raptor is not open source, yet. I didn’t know how to install it from first principles, so I used a binary preview of the installer, obligingly furnished by the Raptor team in Nokia.
This last cheat was unimportant. The Linux Raptor installer will emerge from the contribution pipeline in due course, and in any case I soon afterwards figured out how simple it is to build Raptor on Ubuntu by conventional GNU make methods: the apparatus is all already present and correct in the source.
But the first and second cheats side-stepped real obstacles. Not having an open source compiler that can build the Syborg kernel is hardly less of a problem than not having Syborg. And even when you have built Syborg, without the ROM tools you can’t create a QEMU vm image from it. I didn’t have any idea what size of a task porting the ROM tools for Linux might be.
GCC 4.3 for Symbian
The news from both of these fronts is good. Early in November CodeSourcery, the embedded GCC compiler specialists, finalised a free release of the GCCE (”E” for embedded) v4.3.3 toolchain supporting Symbian on ARM devices. It’s called “G++ Lite for ARM Symbian OS”, and you can get it here, for both Windows and Linux. This release has generated an upsurge of activity in the Symbian community directed at porting the platform to GCCE. The kernel, inevitably, is the prime objective of these efforts. To pull them together, we now have a landing page up on our developer Wiki, The GCCE Toolchain Initiative.
ROM tools for Linux
Building the ROM tools on Linux turns to be a light-to-middling porting job that I’m equal to. I cloned out the build tools package source from its Mercurial repo, http://developer.symbian.org/oss/MCL/sftools/dev/build/, and identified the components I’d need to build a ROM. (Strictly they’re not all ROM tools: there’s a couple of tools in there you need just to build the Syborg kernel – elf2e32, checklib). I studied their MMP files (the Symbian build system’s “build recipe” files), wrote equivalent GNU make files, and then started trying to build the stuff with the the standard GNU/Linux GCC 4.3 toolchain and libraries as shipped with Ubuntu 9.04, plus the Symbian public headers as we shipped them in PDK 3.0.b.
As I encountered failures, I devised fixes and carried on. Mostly these fixes are superficial enough that I can be apply them via the makefiles: they’re to do with GCC 4.3’s more up-to-date and rigorous C++ Standard compliance, versus compilers that the code has previously known; or they’re to do with the fact that the Linux/x86/GCC build configuration isn’t one that the code yet supports. I’ve come across just one component in the build tools thus far ( the sisutils library) that’s thoroughly wedded to Windows APIs and will need thoroughly reworked for portability. But it’s not a library I need to make a ROM.
I’ve built the stuff I need to build a ROM and am familiar enough now with the build tools source to be confident that it will not be difficult to bring up the whole package on Linux. That’s a project I’d naturally like to nudge along, and I’ll get back to it another blog. I’ve committed my work-in-progess to a Mercurial FCL at http://developer.symbian.org/oss/FCL/interim/sftools/linux_build/.
And then again, Syborg
Meantime, I still can’t build a ROM with wholly open source means. Armed with GCCE. QEMU and my ROM tools, I turned to building the Syborg kernel. But that proves not to be a walk in the park. In theory, GCCE will do it. In reality there is a thorny thicket of things that GCCE objects to about the kernel source and I’ve yet to see daylight at the other side of it.
But happily here I have common cause with the brains engaged in the aforementioned GCCE Toolchain Initiative. The thorny problems lie between that compiler and the kernel. Linux isn’t implicated. The Toolchain Initiative is well-connected with the Kernel & Hardware Services team in Nokia, and a GCCE shipshape kernel will surely get committed to our master codeline in the next couple of months, whether or not I do anything to help it. I’ve done a bit to help, by way of committing half-a-dozen “Compiler Compatibility” bugs based on my GCCE kernl-building experience.
Then it will be possible to build and run a Symbian vm by wholly open source means. Or at least it will be possible to start discovering the bugs in my Linux ROM tools.
Another brainwave
Remember the Symbian ideas website? – http://ideas.symbian.org.
I’ve just posted another brilliant idea there, “Create an online Symbian virtual phone configurator”.
But something’s broken in the website. It throws away my formatting and turns my idea into one incomprehensible string.
So here it is the way it was meant to be. If you like it, go and vote for it…
I suggest that we develop an online Symbian virtual phone configurator. The idea is:
- I’m a Symbian developer, anybody from an amateur application developer to a network operator or handset maker.
- I want to have a device with certain characteristics as an early development and testing target, but I want it virtual, so I can integrate it with my PC hosted development environment – like the Symbian Emulator. I especially want it virtual if devices like the one I want aren’t yet on the market.
- I go to the Symbian Virtual Phone Configurator website and there I configure a device that meets my spec from a rich set of parameters, very much in the way that I can configure a car that I might like to buy at the manufacturer’s website (only more flexible than that).
- When I’ve finished configuring, the site builds a Syborg virtual ROM according to my spec, bundles it with virtual drivers according to my spec, and I download a virtual Syborg device of the kind I want that I can run as a QEMU virtual machine under Windows or Linux.
This is an idea for leveraging the breakthrough represented by Syborg. For less technical readers, Syborg is port of the Symbian kernel to run on the QEMU virtual machine manager. QEMU is open source hardware virtualisation solution that enables virtual ARM devices (such as virtual Symbian phones) to be created and hosted on a PC. With Syborg, those virtual Symbian phones can be built. The combination of Syborg and QEMU is on its way now to becoming Symbian’s standard development kit offering for PC hosted Symbian device emulation. This is a solution that allows a virtual Symbian device to be authentic right down to the ARM machine code, and that makes it a far more realistic target for development and testing than the venerable Symbian Emulator. The Emulator will be phased out of our development kits.
But another great merit of Syborg is the ease and flexibility it offers for specifying the characteristics of Syborg virtual devices. (Strictly this ease and flexibility comes from enhancements that the Syborg team have made to QEMU). It is this that makes it practical to offrer “pick ‘n’ mix” assembly of virtual Symbian devices as a web service.
An obvious refinement is that I can have an account on the site, store my device configurations and packages there, share out access to them.
Another is that I could code novel or superior virtual device drivers and contribute them to the site.
As of now, our development kits will be shipping with a prebuilt Syborg/QEMU ROM image included. This is a textshell ROM image, offering only a commandline interface to a rudimentary device. That’s all we are able to build for Syborg now, but as we enrich the platform we will be faced with burgeoning choices as to the spec of a virtual Symbian device that we ship, We could start shipping lots of different ones.
Better, I suggest, to enable the developers to specify the devices they’re interested in. Such a service could become an evolution lab for Symbian devices. And I think it would be a first of its kind – if we do it first.
Implementing this idea would be a substantial collaboration between our web developers, the Syborg team and build engineers, but I do not think it presents profound technical challenges.
Syborg
I promised more about hardware virtualisation for the Symbian platform. Here’s another serving.
The open source QEMU processor emulator brings us the capability to create ARM virtual machines and run them on Intel x86 PCs. Then, if we install Symbian on a QEMU VM, we’ve got a virtual, PC hosted Symbian environment for development and testing that is superior to the venerable Symbian Emulator from every perspective. We are running with this. Soon, Symbian kits will contain a QEMU VM ROM image as well as well as the Symbian Emulator. It will take time for us to bring the features of the QEMU image up to parity with the Emulator, but the Emulator’s days are numbered.
We don’t get “Symbian on QEMU” for nothing, however. QEMU let’s us create ARM VMs for PC hosts. But we’ve still got to make Symbian run on these QEMU VMs, and that problem is the same one we face in getting Symbian to run on any novel type of device. The fact that the QEMU device is virtual rather than physical makes no difference.
Getting Symbian to run on any novel type of device is a matter of creating a Symbian baseport for that type of device. “Baseport”, in Symbian jargon, means a port of the Symbian base for a specfic device type, where the Symbian base means the kernel and hardware services. With the right baseport underneath, the higher layers of the platform don’t care what the device is like.
Creating a baseport for QEMU ARM devices was the bit of real boffinry in the way of getting Symbian on QEMU. It has been done primarily by Martin Trojer. His baseport is called Syborg, and you can get it now from the Symbian source repo at https://developer.symbian.org/oss/FCL/sf/adaptation/qemu/. We have a Wiki page up about Syborg & QEMU, and you can follow the very active discussion round Syborg on our Tools forum.
Martin’s repo doesn’t just provide the Syborg baseport. It also contains a source fork of QEMU itself, which builds in conventional GNU autotools style: ./configure, make, make install. This fork enhances the offcial QEMU distribution with the flexibility to specify a VM’s peripherals via Python plugins and thus to load a VM definition into QEMU dynamically, rather than having to compile the specification of each available VM into a build of QEMU. Let’s hope Python plugin support is accepted into the official QEMU mainline.
There’s one great plus for QEMU v. the Symbian Emulator I haven’t mentioned yet. Unlike the Emulator, QEMU runs on Linux as well as Windows. That means that if we can build the Syborg baseport on Linux, and build Syborg ROMs on Linux too, we’ll have a Linux hosted virtual Symbian environment and remove the fundamental obstacle that has always stood in the path of Symbian development on Linux.
Both of the remaining prerequisites for that breakthrough – building Syborg on Linux and building Syborg ROMs on Linux – are problems I have attacked in the weeks since Martin Trojer’s contribution went in – and I’ve hacked through solutions for both them on Ubuntu Linux. Here’s a screenshot of the Symbian text shell running in a QEMU window on my Ubuntu laptop.
I have posted some HowTos on the Symbian Tools forum that describe these hack solutions, but they’re already out of date so I won’t give links. I’m going to move the record of my Syborg on Linux recipes onto the Symbian Developer Wiki in the next few days and keep them refreshed there. Expect more Syborg blogs here as I make progress.
Ideas at Symbian
Did you know Symbian runs a “new ideas” website? - http://ideas.symbian.org. This is partly a blurb for it.
Anybody – not just Symbians – can register on the site and become a contributor of ideas, and/or a participant in the discussion of other people’s ideas. And you can vote for or against any of the ideas.
Ideas that attract sufficient support will enter a graduation process leading to expert review and, if expert approval is forthcoming, Symbian will run with the idea.
Even if you haven’t the time or inclination to participate on the site, you can browse it without registering and get an eyeful of the thinking that is going on about how Symbian can innovate, improve and grow on all fronts.
This is also a blurb for an idea that I have just floated on the site. My idea is:
“A Build & Test results portal at developer.symbian.org”
And my pitch is:
“I propose that a new tab should be added to the Developer website menubar with a title like “Build & Test Results”, giving access to a menu like:
- Platform builds
- Package builds
- Platform tests
- Package tests
- Binary Compatibility testsand that these links will lead to pages on which Symbian publishes “piping hot” the results of the relevant build and test operations.
The proposed new tab will provide a front door to the latest information of interest to all concerned with build & test progress and quality, and its history. It will showcase our continuous activity at the coalface – a key vitality demonstrator for open source projects. As we bring up the necessary capabilities it can be extended to embrace metrics and trend analysis.
Implementing this idea will call for work by the Build & Integration and Test teams to make results accessible for web publication, but this has already started. Those efforts should be joined up with a commitment that the rolling build & test results will have a prominent, unified entrance from developer.symbian.org.”
If you concur with that – or indeed if you oppose it – you can sign up to http://ideas.symbian.org in a jiffy and cast your vote.
Look, no hardware!
Historically there have been two basic modes for testing the Symbian platform, or a component of the platform or a Symbian app. You could test on an emulator or you could test on hardware.
Emulator testing and hardware testing.
A Symbian emulator emulates the platform, plus the user interface of some fictional device that runs the platform, all as native software for some other platform (where “some other” will in fact mean “Windows”). So with a Symbian emulator on your PC, you can fire it up and you will find yourself looking at a window that more or less ressembles the front face of a fictional Symbian mobile phone. With your mouse and keyboard you can manipulate the emulator’s input features as if you were fingering the phone, and it will behave convincingly like a Symbian phone. You can then use the emulator for testing the platform or applications running on it.
Testing on hardware means testing on a physical device that runs the platform natively. This might be a consumer device that’s already in the market, or it might be a prototype or a pre-release model of a consumer device, or it might be a reference device, a.k.a development board, upon which consumer devices might be technically based. At Symbian, development boards dominate in hardware testing, because typically we’re testing releases of the platform for which consumer devices have yet to be designed or built.
The old tradeoff.
In favour of emulator testing, you don’t need to get actual devices to do it. And it’s easy to do, compared with hardware testing – especially if you want to automate the testing. After all the whole shebang is just software; it’s all virtual. Hardware testing faces the fundamental bother of getting the thing you want to test onto a device and running. And if the thing is not just an installable app – if it’s part of the platform – that means flashing the ROM. And then if you aspire to do hardware testing that’s automatable and halfway scientific, you will need the device hooked up to a test rig that is lab furniture, not home/office workstation.
Against emulator testing is that fact that it’s fake testing, in quite a serious sense of fake. Not only is the platform not running on a real device, it’s not even running on a virtual device that emulates a real device. It’s just running as an elaborate shim on the native OS, with a user interface similar to that of a device. This degree of fakery is good enough for quite a lot of testing to be done on emulator with quite a lot of confidence in the results. But it also leaves a major blind-spot, where tests turn on the device-interfacing innards of the platform, and there are no such innards. Confidence in all emulator testing is sapped to some extent by the knowledge of the underlying fakery – that the call-stacks never really reach the hardware interfaces they’re supposed to.
And among those hardware interfaces it’s not unreasonable to include the very machine instructions of the target CPU. Symbian runs natively on ARM devices, but all emulators to date have run on x86 Windows. They have to be built with an x86 Windows toolchain distinct from the ARM Symbian toolchain. Thus the emulator is not even expressed in the same machine language as the hardware platform; it isn’t generated by identical compilation and linkage algorithms. In this sense, it is faked at level below programmatic design or control – a level where you have to trust that the minute details don’t matter, but very occasionally they do, and then it takes real boffins real trouble to fathom them.
Supporting that kind of boffinry is one of the overheads bourne by those who make the emulator; but it is a tiny fraction of the whole effort of writing the emulator and maintaining it, building and testing and shipping it. And for those, like us at Symbian, whose business is to build, test and ship development kits that can target both emulator and hardware, supporting the emulator is all told as big a burden as supporting hardware. It’s a monster inventory of binaries; another epic build, with another toolchain. Packaging it for kits is a big deal. It’s a parallel world of technical and operational needs.
The Third Way.
Now there’s an alternative to emulator testing that has all of its advantages and very few of its drawbacks. When the Symbian emulator started life last century, hardware virtualisation was a technology developed only for mainframe environments. Lesser computers did not have the heft to exploit it, so its development for small-iron architectures and operating systems was embryonic. In the last 5 years or so, Moore’s Law has transformed that situation. Hardware virtualisation is now mature for the likes of Intel and ARM architectures, for Windows and Linux, and virtual machine technology is now the burgeoning transformer in business IT.
Hardware virtualisation means emulating a computer’s basic hardware in software; not emulating an operating system, but emulating a machine on which various operating systems can run. There are now slick virtualisation tools, both proprietary and open-source, supported on all popular OSes, that let you specify and create virtual machines that you can run within a window on your real PC. When you have created, say, a VM that emulates a vanilla Intel x86 PC, your virtual machine manager will let you boot it in a window and install a “guest” operating system on it from conventional media. You can install any guest OS that will run on that virtual “hardware” – the same or different from the OS of the host PC. Then, you can have, say, genuine MS Windows running in a window on your Fedora Linux desktop, or vice versa.
“Genuine” is the potent word there. Once we have robust hardware virtualisation of ARM devices on x86 hosts, a Symbian emulator is obselete. For any testing purpose that the emulator formerly answered, you will instead be able to build a genuine Symbian ARM ROM, or take one ready built from a development kit, and run it in an ARM virtual machine, on a PC, with all the convenience of the emulator. “Realism” goes all the way down to the hardware interfaces and the machine instructions you are executing.
There remains a significant sense in which virtual machine based testing is fake: testing the performance or timing of hardware-bound operations on a VM is futile, because only a physical device can yield “true readings” in these respects. So testing on actual hardware can’t be wholly dispensed with by virtualisation, but virtualisation reduces to its theoretical minimum the scope for hardware testing to yield bad news.
QEMU.
I said hardware virtualisation for the ARM architecture is now mature. This comes thanks to the open source QEMU project (http://www.nongnu.org/qemu/). Other virtualisation solutions are bigger names: VMWare (proprietary), Virtualbox (open source), but their offerings are restricted to supporting Intel x86 virtual machines on Intel x86 hosts: they don’t go the distance of emulating the VM’s processor. QEMU does. It supports several permutations of virtual and host processor architecture, including ARM vms on x86 hosts.
For us in Symbian, the prospect of adopting QEMU virtualisation as a test environment for the platform holds out an upside much bigger than just the “higher fidelity” that it allows us to achieve in testing-without-hardware, and to offer the users of our development kits. It means that we can do this without supporting anything as enormously costly as the Symbian emulator to do it. ARM virtualisation on x86 is a technology we very much want; but it’s not “our business”, and we’re by no means the only ones who want it. The whole ARM ecosystem wants it, and QEMU has done it. We’re looking at a case in point of open source industrialisation.
That prospect, however, isn’t cost-free. QEMU supports ARM vms on x86 hosts, but it doesn’t come with ready-made virtual machine templates for the sorts of mobile devices that run Symbian, nor even with ready-made tooling to create such vms with speed and ease. We will need to add value to QEMU to create a Symbian hardware virtualisation solution that is slick for our internal operations and for the users of our kits. That work is already well advanced, and you’ll be hearing more about it.
Succumbing to convergence.
Like most people who were already well grown up when cellular telephony was invented, I missed out on the formation of that deep-seated need to have a mobile phone about my person that characterises younger folk. I am never going to be panicked by the prospect of not being in 24/7 telephonic contact with my friends. I’ve worn a watch since my 11th birthday and, if I’m ever deprived of my watch, I have withdrawal symptoms: I feel ill at ease; I have a nude left arm; I am vaguely vulnerable. I have even known myself to wear my watch when it is not working, to spare myself the discomfort of going watchless. My watch-dependency is sub-rational and, ever since I’ve had a mobile phone that tells the time, it is irrational. Having a watch was grafted into my labile young self-image; having a mobile wasn’t.
Mobile phones have had to inflitrate my habits and needs when they were mature habits and needs, unaided by the sort of emotional bondage that my first wristwatch exerted on me, and it has taken a very long time for them to make deep inroads. I got my first “mobile” phone in 1992. A high-end, light-weight BT model of the day, it weighed 1/2 lb and I carried it, in my rucksack or briefcase, only when I planned to make or receive calls out of reach of a landline. I’ve had a mobile ever since but have never felt the stirrings of a phone nut within me. My first couple of smartphones – a Nokia 7650 in 2002 and a then a 6670 in 2004, the former classic device, the latter a turkey – were actually not mind-alteringly superior to my last “dumb” phone, but twice as big and heavy. They doubled up as low quality cameras, which was useful, and it was helpful that they connected with PCs – especially to get the low-quality pictures out. It wasn’t until I got my current Nokia N98 8GB a year-and-half ago that I started to sense that the device had really transcended its utility niche in my activities and become my pocket life-tool. It was important the that camera was by now as good as my rudimentary skills can well exploit, but far from decisive.
When smartphones were the next big thing, pundits often impatiently and sceptically broached the question of what their “killer app” was going to be and when it would show up. But there was vanishingly little chance of an app showing up that would trump being a mobile phone, which was already old hat. Texting was a good second, but that was also old hat. The vision that the smartphone makers have always hewed to is that killer technologies, or killer synergies of technologies, could be made to converge with mobile telephony as they emerged, and support such rich environments of personal applications that holistically compelling devices could be brought to market, for all sorts of folk. And that is how it is panning out.
There’s no one feature or application on my N95 that gives it its invasive, pervasive influence. It’s the sheer wealth of ever-ready, mobile, connected, personal usefulness that it packs. In this device, convergence achieved the tipping point where the vision started working, for me.
It was part of the vision that convergence could hook me in a way quite different from the way it hooks you. The early combo of phone | camera clearly did it for a lot of us. The phone | camera | jukebox | games-console envelope probably embraces the largest – certainly the most conspicuous and most courted – tribe of converged users now. But it doesn’t embrace me: I’m uninterested in software games and I prefer to live most of my life without a soundtrack. The compelling converged device that the N95 embodies for me is a phone | camera | net-port | location-based information server.
A few weeks ago I went on holiday to my native Ireland. With my laptop and N95, I was fully covered for mobile technology. I knew I was going to spend most of my time with no conventional broadband provision, but my mobile handles that: thanks to an app called JoikuSpot (http://www.joikushop.com/), I can use it as a personal WiFi hotspot anywhere I have mobile coverage. I knew I was going to need an overnight stop or two for short tours in the West, but I didn’t pin myself down with advance bookings: I could sort them out just-in-time on the web, anywhere. On the car I’d hired to collect at Belfast, I declined the GPS option: having Ovi Maps on my mobile would be good enough.
I also planned to keep up my running mileage on holiday. Runners are like that. I’m a cycling enthusiast too, and for both activities my mobile has become my motivational tool. I’ve got Nokia’s Sports Tracker app installed (http://sportstracker.nokia.com/). For running, I get shorts with a bum-pocket where the N95 fits snug. When I’m about to hit the road, I turn on Sports Tracker and it tracks my progress by GPS. When I finish, I stop Sports Tracker and it saves my workout: start-time, duration, distance, average and max speed. For running, I also get calorie consumption. I upload the workout to my page on the Sportstracker website. When I visit my page, I can see the route plotted on Google Maps. I can see the stats, and the graph of my speed over altitude. All my previous efforts are likewise available, and I’ve got cumulative training bar charts by days, weeks, or months. Heart-rate junkies can link Sports Tracker by bluetooth to a Polar HRM belt and upload that data to their web page. I never quite talked myself into shelling out for a dedicated GPS training computer and now I don’t need one: it’s been folded into my phone.
But inside the terminal at Birmingham airport, I discovered that I didn’t have my phone. I pictured myself leaving it behind on the kitchen table in Oxford, when my son phoned me to rendezvous for the trip just as I was about to walk out. (As it transpired 10 days later, it had actually fallen onto the floor of my car when we got out at the airport long-stay. ) I felt my first deep pang of mobile phone deprivation. I had no phone, no alarm clock, no camera, no email, no web, no GPS navigation, maps or location-based information, and no Sports Tracker! I tasted what I suppose is already the clinically identified dread of being offline.
All the same, I kept my head, experienced no impulse to head back to Oxford for my phone and succeeded in enjoying a memorably fun holiday without it. My son, of course, had his own camera phone, and even in a B&B in Donegal there is now likely to be free WiFi for the laptop toting traveller. I found myself capable of relishing my runs on the scenic coastal pathways of Belfast Lough, unbothered by the lack of GPS logging. Convergence has morphed my mobile into a my pocket life-tool, but I can still quickly regress to an unconverged modus vivendi. If I’d discovered that I’d somehow come on holiday without my watch, I’d have had to buy another one at the airport.
Recruitment, Recruitment, Recruitment
Eight test engineers is our target. Exactly 4 months from the off, we’ve hired 2. My eyes have been opened to the magnitude of the recruitment task for a high-tech business trying to crew up (even) 200 of the right people from scratch.
4 months, 2 hires, sounds calamitous. But when you break down the numbers, we’re not doing that badly at converting CVs into bums on seats. Here’s the score.
Our recruitment agency has put up 12 CVs. We wanted to interview 10 of those candidates. 2 of them took other offers before we got the chance. 4 we turned down after interview. We made offers to 4. One of the 4 turned us down on salary. One has still to give us an answer. 2 accepted.
Put like that, the story looks like pretty mainstream experience. It’s to the agency’s credit that we’ve liked the look of nearly all the CVs they put up, and we’ve made offers to half of those we’ve interviewed. We are not being too choosey for our own good. In a couple of cases calendar conjestion meant we weren’t quick enough to get to interview. That always happens and you always kick yourself.
It’s the trickle of promising CVs that explains the paltry bottom line. Could be the agency isn’t looking hard enough, but those folks have been successful specialist recruiters in the IT industry for over 20 years. They’d hardly still be here if they weren’t good at.
You might think that the soaring unemployment rate in recession struck UK – recession struck Europe and world – would have created a queue for every vacancy. And it is unhappily true that I’ve several times seen redundancy as a reason for a candidate’s availability in this recruitment drive. But the supply of credible Symbian OS test engineers was always niche and has not been vastly augmented by the general shedding of labour. And of course the high calibre people are the most securely employed, disproportionately rare among the inflated ranks of job-seekers.
I’ve had a hand in the recruitment of the teams I’ve worked in since the early 90s, but always for enterprises very much larger than Symbian is. I regard myself as a careful recruiter, but I know that stage of fatigue in a recruitment slog where it is conceded that the standard has to be dropped: good enough will have to be good enough. Not feeling near to that concession yet. But I do have a novel appreciation of how much more repugnant it would feel in a small enterprise than it does in a big one. If I am recruiting for an enterprise of a 100 people and you are recruiting for an enterprise of 10,000, then on average, the calibre of each person we respectively hire is going to matter 100 times as much to success of my enterprise as it does for yours. Whoppingly crude arithmetic that, but there’s something in it.
An unwelcome corollary of that is the degree of inhibition you feel, in the small organisation, about hiring green talent. The smaller you are, the heavier weighs the cost of developing that talent up to break-even value to the business; the more urgent it appears that the new hire will be able to hit the ground running.
So is Symbian going to have a graduate intake? Recruiting graduates is a social responsibility for employers like us; and yes we are. Just within the Delivery team, of which Test is a part, we have three graduate traineeships in the works for this year – better than 10% of headcount – and one of them’s coming to me in the test team. They will be heartily welcome, but I need them to be good!
And then, ATS4
When I lasted posted, I had no real idea when ATS4 – the next major release of Nokia’s Automated Test System – might break cover. I said “sometime later this year or early next”. But it broke cover last week.
I’ve been keen to see this and keen to move on from ATS3 in our testing operations, because ATS4 is the release with which ATS goes open source and becomes eligible for distribution to Symbian members and contributors. As an almost inevitable corollary, it also goes multi-platform to Linux as well as Windows: this would be a Linux public debut for the ATS team. ATS is a solution for the large-scale tester, not the lone contributor, but most Symbian members are organisations that will likely have a use for a large-scale testing solution. If we can offer them one that we use in Symbian there’s a big synergy there, and that adds merit to ATS for me.
The manner in which ATS4 made its appearance took me by surprise, as well as the timing. ATS3 releases have reached me by sneaker-net. Nokia drop them on an internal FTP server in Finland. A Nokia test engineer in Friars House, Blackfriars Road, London, downloads the release to upgrade his own test network. Then as a favour, he copies it to a USB stick and walks across the road to the gittering Symbian Symplex and delivers it to me. 21st century or what? I thought I’d get my first sight of ATS4 the same way; but when I last enquired about its ETA, my Nokian contact told me it would be on Sourceforge, and that some was already there.
And so it was. This made me think that the ATS team had maybe raised its open-source game more sharply than I’d reckoned with.
The bit that’s already on Sourceforge isn’t a bit that I was after. It’s a new bit. ATS4, I read in the Sourceforge posting, has become “the Nokia ATS tool family“, and the first family member to appear there is one called ATS4AppModel, billed as “an application flow design tool supporting application specification work, model based testing and test script generation. It provides simple interface to manage complex models.”
This sounds cool and I’ll certainly be giving it a test drive to see what it might do for testing in Symbian, but it’s not the open-source successor to ATS3 that I’ve been waiting for. I enquired again and learned that what I am waiting for is called ATS4Core. ATS4Core looks to be released imminently, because the developers have already got a promissory posting about it on Sourceforge. But the software itself isn’t there yet.
Failing that, I thought I’d try spinning up ATS4AppModel on an Ubuntu 9.04 system just to get the measure of the Linux competence of this first Linux offering from ATS.
There wasn’t any GNU tarball, .deb or .rpm, just a zip archive called ATS4AppModel_2009w24_binaries.zip. This didn’t look at all au fait with the way software is packaged for Linux. But it extracted as a directory, ATS4AppModel, that appeared to require no further installation. It contained only 2 platform-dependent files: ATS4AppModel.bat and ATS4AppModel.sh. The former runs the app on Windows; the latter runs it on Linux. Absolutely everything else was platform-independent. All of the code was Java. Fair enough. The readme.txt said I would need Java JRE 1.6.0 or later, so I installed it from the Sun website, since the packaged JRE for Ubtuntu 9.04 is GNU 1.5.
Unhappily, when I ran ATS4AppModel.sh, things got no further than a class-not-found exception for a Swing class. But when I inspected the script, the problem took only a moment to fix. The script said:
java -jar -Xmx512m ATS4AppModel.jar &
The author had not reckoned with the case – like mine – in which JRE 1.6 is installed, but a pre-existing and older Java is on the PATH. To be sure of picking up the latest and greatest Java you’ve installed, best rely on the JAVA_HOME environment variable, which on my system was indeed pointing at the Sun JRE 1.6.0 executable. I edited the script to say:
$JAVA_HOME/bin/java -jar -Xmx512m ATS4AppModel.jar &
Then ATS4AppModel started up without further ado and let me noodle around pretending to model an app.
Just one other thing betokened an uncertain grasp of Linux. That & at the end of the end of the launch command was evidently an attempt to background the Java process and stop it blocking the console in which it was launched. Well, it will background the process but it won’t stop it blocking the console, and there’s no other reason for backgrounding it. I edited the script again to unblock the console:
nohup $JAVA_HOME/bin/java -jar -Xmx512m ATS4AppModel.jar 1>ATS4AppModel-log.txt 2>&1 &
So. ATS4’s open-source debut fumbles a bit with the Linux armature, but religiously portable programming of the payload seems to save the day. Let’s hope this judgement holds good for ATS4Core.
Introducing ATS3
I was talking last time about an automated testing architecture with one “frontend” and many “backends”. We develop the frontend, which accepts a package drop, extracts the package testing info from it, and despatches it to the backend that this information nominates to do the testing. This backend will be an automated testing solution that somebody has contributed to Symbian.
This plan means that if you want to contribute code to Symbian, and you’ve got a testing solution for it that we don’t have, you have the choice of contributing your testing solution as well, and we’ll consider taking it and supporting it on its merits.
As of now, only one automated testing solution has been contributed. That is Nokia’s ATS3 (Automated Test System V3). In fact it hasn’t even really been contributed yet. We’re getting regular drops of the binary installation, but the code is still not open-source. That will come with ATS4, sometime later this year or early next.
Over the last couple of months I’ve been bringing up ATS3 in Symbian to give us an initial automated test capability. It’s a powerful, flexible and scaleable solution and, by the standards of in-house enterprise development, it’s well done. It’s very responsively supported by Nokia’s ATS development team and they ship a release every 2 – 4 weeks.
ATS is automation at a higher level than specific automated testing applications. It’s really an automated test management system. In ATS terminology, a specific automated testing application is a harness. ATS can automate tests for all of the harnesses that Nokia themselves use, and it also supports a “generic” harness, that can wrap any automated testing application that’s capable of being controlled via a fairly adaptable XML protocol. So harnesses are “pluggable” for ATS, though the plugging is pretty fiddly.
ATS has a client-server-worker architecture. An ATS server accepts test-requests from clients and despatches them to ATS workers for execution. When you install the ATS worker component on a machine, you tell it the DNS name of the server you want it to work for. Then whenever that worker starts up, it will register its availability with that server. This makes load distribution and scaling a piece of cake.
The server’s software stack consists of Java, MySQL, Apache Tomcat and STAF – IBM’s open source System Testing Automation Framework. Both STAF and Tomcat depend on Java. The server controls a MySQL database in which all persistent data is stored. A Tomcat web-app provides a browser based GUI for the server. STAF supports all the server-worker communications.
The worker’s software stack is just Java and STAF. On the client side, you just need your browser and some simple shell scripts for posting test requests to the server.
An ATS worker hosts 1 or more devices. A device is just a set of hardware and/or software resources that support the execution of tests for some harness. So a device could be a a phone, or a hardware reference board, or an installation of a Symbian OS emulator.
The ATS server needs to know about all the devices at its disposal. You define a device by writing a properties file. You register the device with the server by uploading this properties file through the server GUI and binding the device it to the worker machine that hosts it. Then device becomes available whenever that worker is available.
You can request a test run by logging into the server’s Tomcat GUI and manually entering the details. But automation is more to the point. Client-side software can request a test run by posting an action URL to the server web-app. There’s no special client needed to do this. You can do it cURL or anything that’s able to post a URL to an http host.
A test request posted via URL will tell the server the LAN location of a testdrop that specifies the test run you want. A testdrop is a zip package that contains (a) an XML recipe for prepping and executing the test with a specified harness, (b) a manfest of any files that have to be deployed on the device for the test prep, (c) all the files mentioned in the manifest, if any. The server will then retrieve the testdrop, read it, and despatch a test run accordingly.
Optionally, the XML recipe can name a specific device on which the test must be run. In that case the server will queue the test for execution on that device when it becomes available. Otherwise, the server will look at the harness the test drop specifies and queue the test for execution on the first available device that supports this harness. You can also schedule tests for execution at times of your choosing, but I’ll skip that. A test run produces and test log and a report of the test results. These are retained by the server until you delete them. You can browse them or download them.
Altogether there’s a good case for making ATS itself the “frontend” for Symbian’s whole automated testing solution. But I don’t think it;s quite good enough. Getting ATS to run even the simplest test for you, even on an emulator, is a fiddly thing to set up – and you really have to be a inside user on the LAN that hosts the ATS network: I’d like something with simpler needs at least for basic use, and I’d rather our frontend service relied only on the internet, so it could be opened up to the contributor community at large.
This is not to say however that ATS could not be used to do the heavy lifting of automated test management for us, if we develop “simplification layer” to go around it and make that layer live on internet protocols. This idea appeals, because ATS can do everything I think our frontend automation calls for, and we won’t have to maintain it.