Back to Syborg on Linux, with GCCE
In my “Syborg” article I made the point that Syborg – the Symbian kernel port that runs on QEMU – is a breakthrough for Symbian development on Linux. Symbian virtual devices with the Syborg kernel, running in the QEMU virtual machine environment, offer the same emulated Symbian development target under both Windows and Linux. The Symbian Emulator of yore was Windows-bound.
I did some experimental work to show that, with fairly superficial hacks, the Syborg kernel could be built, and a Symbian textshell ROM could be built and run on Ubuntu Linux using the code base of Symbian^2.0.0. I wrote up my findings and posted them on the Symbian developer wiki:
How to build the Syborg/QEMU baseport for Symbian^2.0.0 on Ubuntu Linux
How to build Symbian QEMU on Ubuntu Linux
How to build and run a Syborg/QEMU textshell ROM on Ubuntu Linux
Toward Symbian development by open source means
These efforts represent initial moves toward the goal of making Symbian development doable with wholly open source means.
That goal entails making Symbian development doable on Linux. So knowing that it’s already practical to build and run a rudimentary Symbian vm on Linux is an important encouragement.
Those efforts fell short, however, of demonstrating how to build and run a Symbian vm by wholly open source means. I cheated on three points:
- I didn’t have an open source compiler that could build Syborg. I used ARM’s proprietory RVCT compiler v2.2, for which Symbian has licenses.
- I didn’t build the necessary ROM-building tools for Linux. So far, Symbian kits provide only Windows executables for these tools and though they are open source, they haven’t been ported to Linux. I let Ubuntu execute the Windows executables via the Wine program loader.
- I didn’t have an open source installation of the Symbian build system, sbsv2 (a.k.a. “raptor”). Raptor is also open source, and runs on Linux, but the Linux installer for Raptor is not open source, yet. I didn’t know how to install it from first principles, so I used a binary preview of the installer, obligingly furnished by the Raptor team in Nokia.
This last cheat was unimportant. The Linux Raptor installer will emerge from the contribution pipeline in due course, and in any case I soon afterwards figured out how simple it is to build Raptor on Ubuntu by conventional GNU make methods: the apparatus is all already present and correct in the source.
But the first and second cheats side-stepped real obstacles. Not having an open source compiler that can build the Syborg kernel is hardly less of a problem than not having Syborg. And even when you have built Syborg, without the ROM tools you can’t create a QEMU vm image from it. I didn’t have any idea what size of a task porting the ROM tools for Linux might be.
GCC 4.3 for Symbian
The news from both of these fronts is good. Early in November CodeSourcery, the embedded GCC compiler specialists, finalised a free release of the GCCE (”E” for embedded) v4.3.3 toolchain supporting Symbian on ARM devices. It’s called “G++ Lite for ARM Symbian OS”, and you can get it here, for both Windows and Linux. This release has generated an upsurge of activity in the Symbian community directed at porting the platform to GCCE. The kernel, inevitably, is the prime objective of these efforts. To pull them together, we now have a landing page up on our developer Wiki, The GCCE Toolchain Initiative.
ROM tools for Linux
Building the ROM tools on Linux turns to be a light-to-middling porting job that I’m equal to. I cloned out the build tools package source from its Mercurial repo, http://developer.symbian.org/oss/MCL/sftools/dev/build/, and identified the components I’d need to build a ROM. (Strictly they’re not all ROM tools: there’s a couple of tools in there you need just to build the Syborg kernel – elf2e32, checklib). I studied their MMP files (the Symbian build system’s “build recipe” files), wrote equivalent GNU make files, and then started trying to build the stuff with the the standard GNU/Linux GCC 4.3 toolchain and libraries as shipped with Ubuntu 9.04, plus the Symbian public headers as we shipped them in PDK 3.0.b.
As I encountered failures, I devised fixes and carried on. Mostly these fixes are superficial enough that I can be apply them via the makefiles: they’re to do with GCC 4.3’s more up-to-date and rigorous C++ Standard compliance, versus compilers that the code has previously known; or they’re to do with the fact that the Linux/x86/GCC build configuration isn’t one that the code yet supports. I’ve come across just one component in the build tools thus far ( the sisutils library) that’s thoroughly wedded to Windows APIs and will need thoroughly reworked for portability. But it’s not a library I need to make a ROM.
I’ve built the stuff I need to build a ROM and am familiar enough now with the build tools source to be confident that it will not be difficult to bring up the whole package on Linux. That’s a project I’d naturally like to nudge along, and I’ll get back to it another blog. I’ve committed my work-in-progess to a Mercurial FCL at http://developer.symbian.org/oss/FCL/interim/sftools/linux_build/.
And then again, Syborg
Meantime, I still can’t build a ROM with wholly open source means. Armed with GCCE. QEMU and my ROM tools, I turned to building the Syborg kernel. But that proves not to be a walk in the park. In theory, GCCE will do it. In reality there is a thorny thicket of things that GCCE objects to about the kernel source and I’ve yet to see daylight at the other side of it.
But happily here I have common cause with the brains engaged in the aforementioned GCCE Toolchain Initiative. The thorny problems lie between that compiler and the kernel. Linux isn’t implicated. The Toolchain Initiative is well-connected with the Kernel & Hardware Services team in Nokia, and a GCCE shipshape kernel will surely get committed to our master codeline in the next couple of months, whether or not I do anything to help it. I’ve done a bit to help, by way of committing half-a-dozen “Compiler Compatibility” bugs based on my GCCE kernl-building experience.
Then it will be possible to build and run a Symbian vm by wholly open source means. Or at least it will be possible to start discovering the bugs in my Linux ROM tools.
Syborg
I promised more about hardware virtualisation for the Symbian platform. Here’s another serving.
The open source QEMU processor emulator brings us the capability to create ARM virtual machines and run them on Intel x86 PCs. Then, if we install Symbian on a QEMU VM, we’ve got a virtual, PC hosted Symbian environment for development and testing that is superior to the venerable Symbian Emulator from every perspective. We are running with this. Soon, Symbian kits will contain a QEMU VM ROM image as well as well as the Symbian Emulator. It will take time for us to bring the features of the QEMU image up to parity with the Emulator, but the Emulator’s days are numbered.
We don’t get “Symbian on QEMU” for nothing, however. QEMU let’s us create ARM VMs for PC hosts. But we’ve still got to make Symbian run on these QEMU VMs, and that problem is the same one we face in getting Symbian to run on any novel type of device. The fact that the QEMU device is virtual rather than physical makes no difference.
Getting Symbian to run on any novel type of device is a matter of creating a Symbian baseport for that type of device. “Baseport”, in Symbian jargon, means a port of the Symbian base for a specfic device type, where the Symbian base means the kernel and hardware services. With the right baseport underneath, the higher layers of the platform don’t care what the device is like.
Creating a baseport for QEMU ARM devices was the bit of real boffinry in the way of getting Symbian on QEMU. It has been done primarily by Martin Trojer. His baseport is called Syborg, and you can get it now from the Symbian source repo at https://developer.symbian.org/oss/FCL/sf/adaptation/qemu/. We have a Wiki page up about Syborg & QEMU, and you can follow the very active discussion round Syborg on our Tools forum.
Martin’s repo doesn’t just provide the Syborg baseport. It also contains a source fork of QEMU itself, which builds in conventional GNU autotools style: ./configure, make, make install. This fork enhances the offcial QEMU distribution with the flexibility to specify a VM’s peripherals via Python plugins and thus to load a VM definition into QEMU dynamically, rather than having to compile the specification of each available VM into a build of QEMU. Let’s hope Python plugin support is accepted into the official QEMU mainline.
There’s one great plus for QEMU v. the Symbian Emulator I haven’t mentioned yet. Unlike the Emulator, QEMU runs on Linux as well as Windows. That means that if we can build the Syborg baseport on Linux, and build Syborg ROMs on Linux too, we’ll have a Linux hosted virtual Symbian environment and remove the fundamental obstacle that has always stood in the path of Symbian development on Linux.
Both of the remaining prerequisites for that breakthrough – building Syborg on Linux and building Syborg ROMs on Linux – are problems I have attacked in the weeks since Martin Trojer’s contribution went in – and I’ve hacked through solutions for both them on Ubuntu Linux. Here’s a screenshot of the Symbian text shell running in a QEMU window on my Ubuntu laptop.
I have posted some HowTos on the Symbian Tools forum that describe these hack solutions, but they’re already out of date so I won’t give links. I’m going to move the record of my Syborg on Linux recipes onto the Symbian Developer Wiki in the next few days and keep them refreshed there. Expect more Syborg blogs here as I make progress.
And then, ATS4
When I lasted posted, I had no real idea when ATS4 – the next major release of Nokia’s Automated Test System – might break cover. I said “sometime later this year or early next”. But it broke cover last week.
I’ve been keen to see this and keen to move on from ATS3 in our testing operations, because ATS4 is the release with which ATS goes open source and becomes eligible for distribution to Symbian members and contributors. As an almost inevitable corollary, it also goes multi-platform to Linux as well as Windows: this would be a Linux public debut for the ATS team. ATS is a solution for the large-scale tester, not the lone contributor, but most Symbian members are organisations that will likely have a use for a large-scale testing solution. If we can offer them one that we use in Symbian there’s a big synergy there, and that adds merit to ATS for me.
The manner in which ATS4 made its appearance took me by surprise, as well as the timing. ATS3 releases have reached me by sneaker-net. Nokia drop them on an internal FTP server in Finland. A Nokia test engineer in Friars House, Blackfriars Road, London, downloads the release to upgrade his own test network. Then as a favour, he copies it to a USB stick and walks across the road to the gittering Symbian Symplex and delivers it to me. 21st century or what? I thought I’d get my first sight of ATS4 the same way; but when I last enquired about its ETA, my Nokian contact told me it would be on Sourceforge, and that some was already there.
And so it was. This made me think that the ATS team had maybe raised its open-source game more sharply than I’d reckoned with.
The bit that’s already on Sourceforge isn’t a bit that I was after. It’s a new bit. ATS4, I read in the Sourceforge posting, has become “the Nokia ATS tool family“, and the first family member to appear there is one called ATS4AppModel, billed as “an application flow design tool supporting application specification work, model based testing and test script generation. It provides simple interface to manage complex models.”
This sounds cool and I’ll certainly be giving it a test drive to see what it might do for testing in Symbian, but it’s not the open-source successor to ATS3 that I’ve been waiting for. I enquired again and learned that what I am waiting for is called ATS4Core. ATS4Core looks to be released imminently, because the developers have already got a promissory posting about it on Sourceforge. But the software itself isn’t there yet.
Failing that, I thought I’d try spinning up ATS4AppModel on an Ubuntu 9.04 system just to get the measure of the Linux competence of this first Linux offering from ATS.
There wasn’t any GNU tarball, .deb or .rpm, just a zip archive called ATS4AppModel_2009w24_binaries.zip. This didn’t look at all au fait with the way software is packaged for Linux. But it extracted as a directory, ATS4AppModel, that appeared to require no further installation. It contained only 2 platform-dependent files: ATS4AppModel.bat and ATS4AppModel.sh. The former runs the app on Windows; the latter runs it on Linux. Absolutely everything else was platform-independent. All of the code was Java. Fair enough. The readme.txt said I would need Java JRE 1.6.0 or later, so I installed it from the Sun website, since the packaged JRE for Ubtuntu 9.04 is GNU 1.5.
Unhappily, when I ran ATS4AppModel.sh, things got no further than a class-not-found exception for a Swing class. But when I inspected the script, the problem took only a moment to fix. The script said:
java -jar -Xmx512m ATS4AppModel.jar &
The author had not reckoned with the case – like mine – in which JRE 1.6 is installed, but a pre-existing and older Java is on the PATH. To be sure of picking up the latest and greatest Java you’ve installed, best rely on the JAVA_HOME environment variable, which on my system was indeed pointing at the Sun JRE 1.6.0 executable. I edited the script to say:
$JAVA_HOME/bin/java -jar -Xmx512m ATS4AppModel.jar &
Then ATS4AppModel started up without further ado and let me noodle around pretending to model an app.
Just one other thing betokened an uncertain grasp of Linux. That & at the end of the end of the launch command was evidently an attempt to background the Java process and stop it blocking the console in which it was launched. Well, it will background the process but it won’t stop it blocking the console, and there’s no other reason for backgrounding it. I edited the script again to unblock the console:
nohup $JAVA_HOME/bin/java -jar -Xmx512m ATS4AppModel.jar 1>ATS4AppModel-log.txt 2>&1 &
So. ATS4’s open-source debut fumbles a bit with the Linux armature, but religiously portable programming of the payload seems to save the day. Let’s hope this judgement holds good for ATS4Core.