Look, no hardware!

Historically there have been two basic modes for testing the Symbian platform, or a component of the platform or a Symbian app. You could test on an emulator or you could test on hardware.

Emulator testing and hardware testing.

A Symbian emulator emulates the platform, plus the user interface of some fictional device that runs the platform, all as native software for some other platform (where “some other” will in fact mean “Windows”).  So with a Symbian emulator on your PC, you can fire it up and you will find yourself looking at a window that more or less ressembles the front face of a fictional Symbian mobile phone. With your mouse and keyboard you can manipulate the emulator’s input features as if you were fingering the phone, and it will behave convincingly like a Symbian phone. You can then use the emulator for testing the platform or applications running on it.

Testing on hardware means testing on a physical device that runs the platform natively. This might be a consumer device that’s already in the market, or it might be a prototype or a pre-release model of a consumer device, or it might be a reference device, a.k.a development board, upon which consumer devices might be technically based. At Symbian, development boards dominate in hardware testing, because typically we’re testing releases of the platform for which consumer devices have yet to be designed or built.

The old tradeoff.

In favour of emulator testing, you don’t need to get actual devices to do it. And it’s easy to do, compared with hardware testing – especially if you want to automate the testing. After all the whole shebang is just software; it’s all virtual.  Hardware testing faces the fundamental bother of getting the thing you want to test onto a device and running. And if the thing is not just an installable app – if it’s part of the platform – that means flashing the ROM. And then if you aspire to do hardware testing that’s automatable and halfway scientific, you will need the device hooked up to a test rig that is lab  furniture, not home/office workstation.

Against emulator testing is that fact that it’s fake testing, in quite a serious sense of fake. Not only is the platform not running on a real device, it’s not even running on a virtual device that emulates a real device. It’s just running as an elaborate  shim on the native OS, with a user interface similar to that of a device. This degree of fakery is good enough for quite a lot of testing to be done on emulator with quite a lot of confidence in the results. But it also leaves a major blind-spot, where tests turn on the device-interfacing innards of the platform, and there are no such innards. Confidence in all emulator testing is sapped to some extent by the knowledge of the underlying fakery – that the call-stacks never really reach the hardware interfaces they’re supposed to.

And among those hardware interfaces it’s not unreasonable to include the very machine instructions of the target CPU. Symbian runs natively on ARM devices, but all emulators to date have run on x86 Windows. They have to be built with an x86 Windows toolchain distinct from the ARM Symbian toolchain. Thus the emulator is not even expressed in the same machine language as the hardware platform; it isn’t generated by identical compilation and linkage algorithms. In this sense, it is faked at level below programmatic design or control – a level where you have to trust that the minute details don’t matter, but very occasionally they do, and then it takes real boffins real trouble to fathom them.

Supporting that kind of boffinry is one of the overheads bourne by those who make the emulator; but it is a tiny fraction of the whole effort of writing the emulator and maintaining it, building and testing and shipping it. And for those, like us at Symbian, whose business is to build, test and ship development kits that can target both emulator and hardware, supporting the emulator is all told as big a burden as supporting hardware. It’s a monster inventory of binaries; another epic build, with another toolchain. Packaging it for kits is a big deal. It’s a parallel world of technical and operational needs.

The Third Way.

Now there’s an alternative to emulator testing that has all of its advantages and very few of its drawbacks. When the Symbian emulator started life last century, hardware virtualisation was a technology developed only for mainframe environments. Lesser computers did not have the heft to exploit it, so its development for small-iron architectures and operating systems was embryonic. In the last 5 years or so, Moore’s Law has transformed that situation. Hardware virtualisation is now mature for the likes of Intel and ARM architectures, for Windows and Linux, and virtual machine technology is now the burgeoning transformer in business IT.

Hardware virtualisation means emulating a computer’s basic hardware in software; not emulating an operating system, but emulating a machine on which various operating systems can run.  There are now slick virtualisation tools, both proprietary and open-source,  supported on all popular OSes, that let you specify and create virtual machines that you can run within a window on your real PC.  When you have created, say, a VM that emulates a vanilla Intel x86  PC, your virtual machine manager will let you boot it in a window and install a “guest” operating system on it from conventional media. You can install any guest OS that will run on that virtual “hardware” – the same or different from the OS of the host PC.  Then, you can have, say,  genuine MS Windows running in a window on your Fedora Linux desktop, or vice versa.

“Genuine” is the potent word there. Once we have robust hardware virtualisation of ARM devices on x86 hosts, a Symbian emulator is obselete. For any testing purpose that the emulator formerly answered, you will instead be able to build a genuine Symbian ARM ROM, or take one ready built from a development kit,  and run it in an ARM virtual machine,  on a PC, with all the convenience of the emulator. “Realism” goes all the way down to the hardware interfaces and the machine instructions you are executing.

There remains a significant sense in which virtual machine based testing is fake: testing the performance or timing of hardware-bound operations on a VM is futile, because only a physical device can yield “true readings” in these respects.  So testing on actual hardware can’t be wholly dispensed with by virtualisation, but virtualisation reduces to its theoretical minimum the scope for hardware testing to yield bad news.

QEMU.

I said hardware virtualisation for the ARM architecture is now mature. This comes thanks to the open source QEMU project (http://www.nongnu.org/qemu/). Other virtualisation solutions are bigger names: VMWare (proprietary), Virtualbox (open source), but their offerings are restricted to supporting Intel x86 virtual machines on Intel x86 hosts: they don’t go the distance of emulating the VM’s processor. QEMU does. It supports several permutations of virtual and host processor architecture, including ARM vms on x86 hosts.

For us in Symbian, the prospect of adopting QEMU virtualisation as a test environment for the platform holds out an upside much bigger than just the “higher fidelity” that it allows us to achieve in testing-without-hardware, and to offer the users of our development kits. It means that we can do this without supporting anything as enormously costly as the Symbian emulator to do it.  ARM virtualisation on x86 is a technology we very much want; but it’s not “our business”, and we’re by no means the only ones who want it. The whole ARM ecosystem wants it, and QEMU has done it. We’re looking at a case in point of open source industrialisation.

That prospect, however, isn’t cost-free. QEMU supports ARM vms on x86 hosts, but it doesn’t come with ready-made virtual machine templates for the sorts of mobile devices that run Symbian, nor even with ready-made tooling to create such vms with speed and ease. We will need to add value to QEMU to create a Symbian hardware virtualisation solution that is slick for our internal operations and for the users of our kits. That work is already well advanced, and you’ll be hearing more about it.