Community-driven software QA. My quest for inspiration.
Scene-setter for blogs to come.
I haven’t mentioned since blog #1 that I’m the Test Lead at Symbian. Nine months into the job, it’s sunk in that what it must mainly be about is enabling. Enabling the Symbian community to forge the testing capability it needs.
Probably I’ve been slow to digest this insight because of my prior CV. Until I started this job, for going on 5 years I’d led the team that maintained the Developer Build & Test System (DABS) in Symbian’s corporate precursor, Symbian Software Ltd (call it SSL). DABS was the nightly build and test system shared by the 30 or so software development teams that made up the software engineering force in SSL. In SSL, naturally, it went without saying that we test our software to shippable quality, all by ourselves before anybody else gets a look at it. And DABS bore most of that testing effort.
Now, I can see that I my professional mindset needs reset. So I’ve been looking at Linux distros to see what I – and Symbian – can learn about successful test strategies for a non-profit provider of an open-source personal computing platform; foremost, to learn how these projects are able to harness community resources for their testing operations and the ways in which they do it.
I looked to Linux distributions because they are only sustained examples of successful non-profit providers of open-source personal computing platforms and are exemplary of community powered software production at its most accomplished and ambitious. Symbian by its raison d’être faces the same challenges that face a Linux distro. (But not only the same challenges; a qualification that might be crucial).
I use “non-profit” there in a relaxed sense. I mean that the platform is provided at no charge. All of the most popular Linux distros now – say the distrowatch top 5 – have some sort of corporate backing that turns a profit from professional services and support for the free platform, or by selling and supporting a value-added down-stream spin. Symbian has profit-motivated corporate backing too – our funding members – but our platform is provided at no charge, and nothing is success for Symbian but community-built success.
I’ve studied the web-presenses of leading Linux distros – Debian, Fedora, Mandriva, openSUSE, Ubuntu (impartially alphabetical order here). I’ve peered around the online resources and activities that lie behind. I’ve made some contacts who are in roles like mine, and been welcomed to question what they do, and how and why. (Debian, I must hasten to emphasise, has no corporate stakeholders: it is a pure community project).
I’ve taken away a rich load of perceptions and learning points about community-driven software testing and QA for which I want to get the attention of Symbian, its community and ecosystem. I’ll be blogging on this theme. Not all of my perceptions are of beckoning opportunities for Symbian to catch up. Some of them are of daunting impediments for Symbian catching up.
I’ve also taken away a sense of obligation to reciprocate the openess of the projects I’ve researched. I’m thinking: In the light of everything I’ve learned, what do I know about testing and QA that could be useful to these and other open-source projects? I’ll blog on that theme too.
Look, no hardware!
Historically there have been two basic modes for testing the Symbian platform, or a component of the platform or a Symbian app. You could test on an emulator or you could test on hardware.
Emulator testing and hardware testing.
A Symbian emulator emulates the platform, plus the user interface of some fictional device that runs the platform, all as native software for some other platform (where “some other” will in fact mean “Windows”). So with a Symbian emulator on your PC, you can fire it up and you will find yourself looking at a window that more or less ressembles the front face of a fictional Symbian mobile phone. With your mouse and keyboard you can manipulate the emulator’s input features as if you were fingering the phone, and it will behave convincingly like a Symbian phone. You can then use the emulator for testing the platform or applications running on it.
Testing on hardware means testing on a physical device that runs the platform natively. This might be a consumer device that’s already in the market, or it might be a prototype or a pre-release model of a consumer device, or it might be a reference device, a.k.a development board, upon which consumer devices might be technically based. At Symbian, development boards dominate in hardware testing, because typically we’re testing releases of the platform for which consumer devices have yet to be designed or built.
The old tradeoff.
In favour of emulator testing, you don’t need to get actual devices to do it. And it’s easy to do, compared with hardware testing – especially if you want to automate the testing. After all the whole shebang is just software; it’s all virtual. Hardware testing faces the fundamental bother of getting the thing you want to test onto a device and running. And if the thing is not just an installable app – if it’s part of the platform – that means flashing the ROM. And then if you aspire to do hardware testing that’s automatable and halfway scientific, you will need the device hooked up to a test rig that is lab furniture, not home/office workstation.
Against emulator testing is that fact that it’s fake testing, in quite a serious sense of fake. Not only is the platform not running on a real device, it’s not even running on a virtual device that emulates a real device. It’s just running as an elaborate shim on the native OS, with a user interface similar to that of a device. This degree of fakery is good enough for quite a lot of testing to be done on emulator with quite a lot of confidence in the results. But it also leaves a major blind-spot, where tests turn on the device-interfacing innards of the platform, and there are no such innards. Confidence in all emulator testing is sapped to some extent by the knowledge of the underlying fakery – that the call-stacks never really reach the hardware interfaces they’re supposed to.
And among those hardware interfaces it’s not unreasonable to include the very machine instructions of the target CPU. Symbian runs natively on ARM devices, but all emulators to date have run on x86 Windows. They have to be built with an x86 Windows toolchain distinct from the ARM Symbian toolchain. Thus the emulator is not even expressed in the same machine language as the hardware platform; it isn’t generated by identical compilation and linkage algorithms. In this sense, it is faked at level below programmatic design or control – a level where you have to trust that the minute details don’t matter, but very occasionally they do, and then it takes real boffins real trouble to fathom them.
Supporting that kind of boffinry is one of the overheads bourne by those who make the emulator; but it is a tiny fraction of the whole effort of writing the emulator and maintaining it, building and testing and shipping it. And for those, like us at Symbian, whose business is to build, test and ship development kits that can target both emulator and hardware, supporting the emulator is all told as big a burden as supporting hardware. It’s a monster inventory of binaries; another epic build, with another toolchain. Packaging it for kits is a big deal. It’s a parallel world of technical and operational needs.
The Third Way.
Now there’s an alternative to emulator testing that has all of its advantages and very few of its drawbacks. When the Symbian emulator started life last century, hardware virtualisation was a technology developed only for mainframe environments. Lesser computers did not have the heft to exploit it, so its development for small-iron architectures and operating systems was embryonic. In the last 5 years or so, Moore’s Law has transformed that situation. Hardware virtualisation is now mature for the likes of Intel and ARM architectures, for Windows and Linux, and virtual machine technology is now the burgeoning transformer in business IT.
Hardware virtualisation means emulating a computer’s basic hardware in software; not emulating an operating system, but emulating a machine on which various operating systems can run. There are now slick virtualisation tools, both proprietary and open-source, supported on all popular OSes, that let you specify and create virtual machines that you can run within a window on your real PC. When you have created, say, a VM that emulates a vanilla Intel x86 PC, your virtual machine manager will let you boot it in a window and install a “guest” operating system on it from conventional media. You can install any guest OS that will run on that virtual “hardware” – the same or different from the OS of the host PC. Then, you can have, say, genuine MS Windows running in a window on your Fedora Linux desktop, or vice versa.
“Genuine” is the potent word there. Once we have robust hardware virtualisation of ARM devices on x86 hosts, a Symbian emulator is obselete. For any testing purpose that the emulator formerly answered, you will instead be able to build a genuine Symbian ARM ROM, or take one ready built from a development kit, and run it in an ARM virtual machine, on a PC, with all the convenience of the emulator. “Realism” goes all the way down to the hardware interfaces and the machine instructions you are executing.
There remains a significant sense in which virtual machine based testing is fake: testing the performance or timing of hardware-bound operations on a VM is futile, because only a physical device can yield “true readings” in these respects. So testing on actual hardware can’t be wholly dispensed with by virtualisation, but virtualisation reduces to its theoretical minimum the scope for hardware testing to yield bad news.
QEMU.
I said hardware virtualisation for the ARM architecture is now mature. This comes thanks to the open source QEMU project (http://www.nongnu.org/qemu/). Other virtualisation solutions are bigger names: VMWare (proprietary), Virtualbox (open source), but their offerings are restricted to supporting Intel x86 virtual machines on Intel x86 hosts: they don’t go the distance of emulating the VM’s processor. QEMU does. It supports several permutations of virtual and host processor architecture, including ARM vms on x86 hosts.
For us in Symbian, the prospect of adopting QEMU virtualisation as a test environment for the platform holds out an upside much bigger than just the “higher fidelity” that it allows us to achieve in testing-without-hardware, and to offer the users of our development kits. It means that we can do this without supporting anything as enormously costly as the Symbian emulator to do it. ARM virtualisation on x86 is a technology we very much want; but it’s not “our business”, and we’re by no means the only ones who want it. The whole ARM ecosystem wants it, and QEMU has done it. We’re looking at a case in point of open source industrialisation.
That prospect, however, isn’t cost-free. QEMU supports ARM vms on x86 hosts, but it doesn’t come with ready-made virtual machine templates for the sorts of mobile devices that run Symbian, nor even with ready-made tooling to create such vms with speed and ease. We will need to add value to QEMU to create a Symbian hardware virtualisation solution that is slick for our internal operations and for the users of our kits. That work is already well advanced, and you’ll be hearing more about it.
Recruitment, Recruitment, Recruitment
Eight test engineers is our target. Exactly 4 months from the off, we’ve hired 2. My eyes have been opened to the magnitude of the recruitment task for a high-tech business trying to crew up (even) 200 of the right people from scratch.
4 months, 2 hires, sounds calamitous. But when you break down the numbers, we’re not doing that badly at converting CVs into bums on seats. Here’s the score.
Our recruitment agency has put up 12 CVs. We wanted to interview 10 of those candidates. 2 of them took other offers before we got the chance. 4 we turned down after interview. We made offers to 4. One of the 4 turned us down on salary. One has still to give us an answer. 2 accepted.
Put like that, the story looks like pretty mainstream experience. It’s to the agency’s credit that we’ve liked the look of nearly all the CVs they put up, and we’ve made offers to half of those we’ve interviewed. We are not being too choosey for our own good. In a couple of cases calendar conjestion meant we weren’t quick enough to get to interview. That always happens and you always kick yourself.
It’s the trickle of promising CVs that explains the paltry bottom line. Could be the agency isn’t looking hard enough, but those folks have been successful specialist recruiters in the IT industry for over 20 years. They’d hardly still be here if they weren’t good at.
You might think that the soaring unemployment rate in recession struck UK – recession struck Europe and world – would have created a queue for every vacancy. And it is unhappily true that I’ve several times seen redundancy as a reason for a candidate’s availability in this recruitment drive. But the supply of credible Symbian OS test engineers was always niche and has not been vastly augmented by the general shedding of labour. And of course the high calibre people are the most securely employed, disproportionately rare among the inflated ranks of job-seekers.
I’ve had a hand in the recruitment of the teams I’ve worked in since the early 90s, but always for enterprises very much larger than Symbian is. I regard myself as a careful recruiter, but I know that stage of fatigue in a recruitment slog where it is conceded that the standard has to be dropped: good enough will have to be good enough. Not feeling near to that concession yet. But I do have a novel appreciation of how much more repugnant it would feel in a small enterprise than it does in a big one. If I am recruiting for an enterprise of a 100 people and you are recruiting for an enterprise of 10,000, then on average, the calibre of each person we respectively hire is going to matter 100 times as much to success of my enterprise as it does for yours. Whoppingly crude arithmetic that, but there’s something in it.
An unwelcome corollary of that is the degree of inhibition you feel, in the small organisation, about hiring green talent. The smaller you are, the heavier weighs the cost of developing that talent up to break-even value to the business; the more urgent it appears that the new hire will be able to hit the ground running.
So is Symbian going to have a graduate intake? Recruiting graduates is a social responsibility for employers like us; and yes we are. Just within the Delivery team, of which Test is a part, we have three graduate traineeships in the works for this year – better than 10% of headcount – and one of them’s coming to me in the test team. They will be heartily welcome, but I need them to be good!
And then, ATS4
When I lasted posted, I had no real idea when ATS4 – the next major release of Nokia’s Automated Test System – might break cover. I said “sometime later this year or early next”. But it broke cover last week.
I’ve been keen to see this and keen to move on from ATS3 in our testing operations, because ATS4 is the release with which ATS goes open source and becomes eligible for distribution to Symbian members and contributors. As an almost inevitable corollary, it also goes multi-platform to Linux as well as Windows: this would be a Linux public debut for the ATS team. ATS is a solution for the large-scale tester, not the lone contributor, but most Symbian members are organisations that will likely have a use for a large-scale testing solution. If we can offer them one that we use in Symbian there’s a big synergy there, and that adds merit to ATS for me.
The manner in which ATS4 made its appearance took me by surprise, as well as the timing. ATS3 releases have reached me by sneaker-net. Nokia drop them on an internal FTP server in Finland. A Nokia test engineer in Friars House, Blackfriars Road, London, downloads the release to upgrade his own test network. Then as a favour, he copies it to a USB stick and walks across the road to the gittering Symbian Symplex and delivers it to me. 21st century or what? I thought I’d get my first sight of ATS4 the same way; but when I last enquired about its ETA, my Nokian contact told me it would be on Sourceforge, and that some was already there.
And so it was. This made me think that the ATS team had maybe raised its open-source game more sharply than I’d reckoned with.
The bit that’s already on Sourceforge isn’t a bit that I was after. It’s a new bit. ATS4, I read in the Sourceforge posting, has become “the Nokia ATS tool family“, and the first family member to appear there is one called ATS4AppModel, billed as “an application flow design tool supporting application specification work, model based testing and test script generation. It provides simple interface to manage complex models.”
This sounds cool and I’ll certainly be giving it a test drive to see what it might do for testing in Symbian, but it’s not the open-source successor to ATS3 that I’ve been waiting for. I enquired again and learned that what I am waiting for is called ATS4Core. ATS4Core looks to be released imminently, because the developers have already got a promissory posting about it on Sourceforge. But the software itself isn’t there yet.
Failing that, I thought I’d try spinning up ATS4AppModel on an Ubuntu 9.04 system just to get the measure of the Linux competence of this first Linux offering from ATS.
There wasn’t any GNU tarball, .deb or .rpm, just a zip archive called ATS4AppModel_2009w24_binaries.zip. This didn’t look at all au fait with the way software is packaged for Linux. But it extracted as a directory, ATS4AppModel, that appeared to require no further installation. It contained only 2 platform-dependent files: ATS4AppModel.bat and ATS4AppModel.sh. The former runs the app on Windows; the latter runs it on Linux. Absolutely everything else was platform-independent. All of the code was Java. Fair enough. The readme.txt said I would need Java JRE 1.6.0 or later, so I installed it from the Sun website, since the packaged JRE for Ubtuntu 9.04 is GNU 1.5.
Unhappily, when I ran ATS4AppModel.sh, things got no further than a class-not-found exception for a Swing class. But when I inspected the script, the problem took only a moment to fix. The script said:
java -jar -Xmx512m ATS4AppModel.jar &
The author had not reckoned with the case – like mine – in which JRE 1.6 is installed, but a pre-existing and older Java is on the PATH. To be sure of picking up the latest and greatest Java you’ve installed, best rely on the JAVA_HOME environment variable, which on my system was indeed pointing at the Sun JRE 1.6.0 executable. I edited the script to say:
$JAVA_HOME/bin/java -jar -Xmx512m ATS4AppModel.jar &
Then ATS4AppModel started up without further ado and let me noodle around pretending to model an app.
Just one other thing betokened an uncertain grasp of Linux. That & at the end of the end of the launch command was evidently an attempt to background the Java process and stop it blocking the console in which it was launched. Well, it will background the process but it won’t stop it blocking the console, and there’s no other reason for backgrounding it. I edited the script again to unblock the console:
nohup $JAVA_HOME/bin/java -jar -Xmx512m ATS4AppModel.jar 1>ATS4AppModel-log.txt 2>&1 &
So. ATS4’s open-source debut fumbles a bit with the Linux armature, but religiously portable programming of the payload seems to save the day. Let’s hope this judgement holds good for ATS4Core.
Introducing ATS3
I was talking last time about an automated testing architecture with one “frontend” and many “backends”. We develop the frontend, which accepts a package drop, extracts the package testing info from it, and despatches it to the backend that this information nominates to do the testing. This backend will be an automated testing solution that somebody has contributed to Symbian.
This plan means that if you want to contribute code to Symbian, and you’ve got a testing solution for it that we don’t have, you have the choice of contributing your testing solution as well, and we’ll consider taking it and supporting it on its merits.
As of now, only one automated testing solution has been contributed. That is Nokia’s ATS3 (Automated Test System V3). In fact it hasn’t even really been contributed yet. We’re getting regular drops of the binary installation, but the code is still not open-source. That will come with ATS4, sometime later this year or early next.
Over the last couple of months I’ve been bringing up ATS3 in Symbian to give us an initial automated test capability. It’s a powerful, flexible and scaleable solution and, by the standards of in-house enterprise development, it’s well done. It’s very responsively supported by Nokia’s ATS development team and they ship a release every 2 – 4 weeks.
ATS is automation at a higher level than specific automated testing applications. It’s really an automated test management system. In ATS terminology, a specific automated testing application is a harness. ATS can automate tests for all of the harnesses that Nokia themselves use, and it also supports a “generic” harness, that can wrap any automated testing application that’s capable of being controlled via a fairly adaptable XML protocol. So harnesses are “pluggable” for ATS, though the plugging is pretty fiddly.
ATS has a client-server-worker architecture. An ATS server accepts test-requests from clients and despatches them to ATS workers for execution. When you install the ATS worker component on a machine, you tell it the DNS name of the server you want it to work for. Then whenever that worker starts up, it will register its availability with that server. This makes load distribution and scaling a piece of cake.
The server’s software stack consists of Java, MySQL, Apache Tomcat and STAF – IBM’s open source System Testing Automation Framework. Both STAF and Tomcat depend on Java. The server controls a MySQL database in which all persistent data is stored. A Tomcat web-app provides a browser based GUI for the server. STAF supports all the server-worker communications.
The worker’s software stack is just Java and STAF. On the client side, you just need your browser and some simple shell scripts for posting test requests to the server.
An ATS worker hosts 1 or more devices. A device is just a set of hardware and/or software resources that support the execution of tests for some harness. So a device could be a a phone, or a hardware reference board, or an installation of a Symbian OS emulator.
The ATS server needs to know about all the devices at its disposal. You define a device by writing a properties file. You register the device with the server by uploading this properties file through the server GUI and binding the device it to the worker machine that hosts it. Then device becomes available whenever that worker is available.
You can request a test run by logging into the server’s Tomcat GUI and manually entering the details. But automation is more to the point. Client-side software can request a test run by posting an action URL to the server web-app. There’s no special client needed to do this. You can do it cURL or anything that’s able to post a URL to an http host.
A test request posted via URL will tell the server the LAN location of a testdrop that specifies the test run you want. A testdrop is a zip package that contains (a) an XML recipe for prepping and executing the test with a specified harness, (b) a manfest of any files that have to be deployed on the device for the test prep, (c) all the files mentioned in the manifest, if any. The server will then retrieve the testdrop, read it, and despatch a test run accordingly.
Optionally, the XML recipe can name a specific device on which the test must be run. In that case the server will queue the test for execution on that device when it becomes available. Otherwise, the server will look at the harness the test drop specifies and queue the test for execution on the first available device that supports this harness. You can also schedule tests for execution at times of your choosing, but I’ll skip that. A test run produces and test log and a report of the test results. These are retained by the server until you delete them. You can browse them or download them.
Altogether there’s a good case for making ATS itself the “frontend” for Symbian’s whole automated testing solution. But I don’t think it;s quite good enough. Getting ATS to run even the simplest test for you, even on an emulator, is a fiddly thing to set up – and you really have to be a inside user on the LAN that hosts the ATS network: I’d like something with simpler needs at least for basic use, and I’d rather our frontend service relied only on the internet, so it could be opened up to the contributor community at large.
This is not to say however that ATS could not be used to do the heavy lifting of automated test management for us, if we develop “simplification layer” to go around it and make that layer live on internet protocols. This idea appeals, because ATS can do everything I think our frontend automation calls for, and we won’t have to maintain it.
Package testing, resumed
I left the topic of package testing dangling last time. Detecting bugs before we commit them to the platform is the foremost goal of our testing operations, and nearly all of that goal falls on package testing.
We need the package owners to give us the tests that will qualify a package drop for platform commit. They’ve got the competence and resource to develop the kind of no-kidding tests needed for this quality gate; we haven’t. We’ll expect a package drop to have passed its own tests when we get it; but we’ll want run them again to be sure. The baseline platform build that the drop was tested against might be older than the latest one we’ve reached.
If we haven’t got the capability in-house to shoulder package testing development ourselves, how are we going to know if the package tests we get are good enough? Well we’ll have the expertise to tell if tests that a package comes with are obviously inadequate. But if they’re inadequate and it’s not obvious, it will become apparent because the package gets a bad record for being at the bottom of platform bugs that turn up down the line. Then we’ll need a conversation with the package owner about the quality of those tests.
I want a package to contain the recipe and ingredients for testing it, in automatable form. Package drops will get built automatically. If a build succeeds, I want the automation to carry right on and run the package tests. In GNU autotools terms, the package testing will be a make check step.
We need to define the way in which the recipe and ingredients for package testing can be expressed in the package, so that our automation can understand it. Our automation isn’t actually autotools based; it’s ANT-based. So the package testing recipes could be ANT-files, if we think its fair enough to insist that package owners will also use ANT – and therefore Java – for their own testing. That’s not a trifling committment, so some XML protocol of our own devising, with lighter baggage, could be the answer. An open question.
But the problem of how the package testing recipes can be expressed in an automatable fashion can’t be isolated from the question of the automation technology and infraststructure. To push the culinary metaphor, it’s great if we can understand the instruction “Put the tomatoes in the blender”; but we need a blender to carry it out. And if we tell the world “We’ve got some blenders. We’ve got no pressure cookers”, then anybody who’s got a pressure cooker but no blender can’t send us recipes to cook.
As well as defining how package testing recipes can be expressed, we’ll have to specify the testing technologies at our disposal, so that contributors can give us recipes we’re able to carry out. We are not going to be in the business or originating new testing technologies. The ones at our disposal are going to be ones that are out there open-source already, or else ones that are contributed to Symbian (in which case they’ll be open-source once contributed).
Any automated testing technology we get is going to come with its own provisions about how to express the recipe and ingredients for a test. But I don’t want to bind us to a single testing technology by standardising on the test specification protocol that it lays out. If a prospective contributor is using a testing technology that we don’t support, I don’t want to be telling them there’s nothing doing unless they adopt one that we do support. I want the option that they can contribute their testing technology to us, and that we can hook it up and add it to our repertoire without a whole lot of disruption. I want this extensibility built into our testing automation.
So I’m thinking of one frontend, potentially many backends; like compiler architecture. The automation that we develop here should be a pretty thin despatching layer that can identify the type of backend a package elects to be tested with, and hand it off to a backend of that type to do the real work. The backends will be contributed technologies and resources. The package testing recipes should be expressed in a frontend protocol that we define, and they’ll be “meta-recipes” essentially like this:
- Recipe: Test this package with Gnomotest
- Ingredients: One Gnomotest test specification, from package.
A thumbnail of the Symbian platform lifecycle.
I see that some scene setting articles will be in order to ground this blog. Here’s my first.
What is the process by which the Symbian platform will evolve? – It will be a cycle within a cycle within a cycle.
(I’m talking in the future tense because right now we’re still getting this act together. Which also means that I might have to correct what I say here later on, but I’m pretty sure this won’t be drastically wrong.)
The Symbian platform is made up of packages, like a Linux distribution. Like a Linux distribution too, we (Symbian) own the platform, but we don’t own the packages. They are owned by upstream contributors, who could be any person or organisation. The platform will evolve by the contribution of new packages, or new releases of old ones , and occasionally the retirement of packages. We’ll be geared to deal with a daily inflow of fresh package drops.
The longest cycle is our official release cycle. We’ll make an official release of the platform every 6 months. A release will deliver a planned set of new features or improvements (and possibly removals of obselete features). An official release will be for manufacturers to build devices on and developers to build apps on.
Inside the official release cycle comes our fortnightly release cycle. Every fortnight we’ll make a public release of the platform that represents the most stable, usable drop of the platform that we’ve achieved en route to the next official release. A fortnightly release will be deeply tested. It will let device makers and developers take stock of how we’re getting on, allow them to keep their own developments in step with ours, and let them find bugs or gaps to feed back for fixing.
Inside the fortnightly release cycle comes our daily build and test cycle. This cycle has two threads that will run in parallel: platform build & test, and package build & test.
Every day – at least every day when any change has been comitted to the platform – we will build the whole lot, to make dead sure that we still can. We’ll also test the daily builds lightly, to make sure we can at least still boot a device or an emulator and that it’s not useless when we do. That’s the daily platform build & test thread.
The essential function of the daily platform builds will to generate every day a rev of the platform that’s stable enough to serve as the baseline build for the second thread: building and testing the daily inflow of package drops. We’ll publish the daily builds, including logs and reports so that package owners can see the latest status and get hold of the latest good baseline to build their packages against.
When a package owner sends us a drop, the critical question for us is whether it builds and tests successfully in the context of the latest good rev of the latest good daily platform build that we’ve got. So we’ll build the fresh package source together with the unchanged remainder of that baseline build. If the build is good, we’ll then go on to test it deeply. If the package also tests successfully, then we’ll commit the fresh package source to our mainline platform repository. That night, the whole platform will be rebuilt from that repository, with fresh source from the package drops that have been comitted that day. All being well, we get a new good baseline to publish the next morning, and so it goes…
All not being well, we’ve got build breaks or we’ve got test failures in the platform. We’ll raise defects to the owners of the faulty packages, and our baseline will mark time until the fixes come in.
It’s the daily cycle, of course, that will set our pace. Keeping good builds rolling out steadily is what contributors need from us to keep good new code rolling safely in.