March Madness
Someone once told me: “[Your] release procedures are the most primitive and barbaric I’ve ever encountered.”
After thinking about it for awhile (and getting over the initial sting of the commentary), I came to a couple of conclusions:
First, the person providing the critique had limited experience with… well… a complicated release process, one that involves releasing builds in 40+ locales and delivering updates to the desktops1 of millions of users. This person did have a lot of experience with release processes that he had [the fortune] of being able to design (and implement!) himself, and therefore seldom had to address the issues related to legacy- and/or poorly documented-infrastructure that we must deal with every day2.
Second, in some very real sense, he’s right.
Our current release process, which is the fourth or fifth generation of, from what I can tell, is approximately the third or fourth mostly-completely-different version of a Release Process ™ that we’ve used. It is way more manual3 than it needs to be, has a number of especially-retarded-if-you-don’t-happen-to-know-the-history steps and provisions, encapsulates processes in unnatural-and-hard-to-understand ways, generally takes too long4, and requires an extreme amount of mostly-constant-while-it’s-going-on focus.
On the charitable side it… well… addresses the requirements necessary to release 40+ locales, on three major platforms, including automated updates, for… millions of users.
At the beginning of last October, rhelmer, after having done a number of releases, began looking at ways to automate the current Firefox/Thunderbird release process, warts and all.
He, along with help from others, have made great strides in creating a solid framework for handling the various steps involved in putting out a release. A great side-effect of his work is that it has put our release process—warts and all—into the public repository with all our other code, so others can benefit5… and hopefully contribute.
One of our goals for the month of March is to knit together the release automation so that we can realize the seemingly-never-to-materialize-mirage of complete, end-to-end automation.
To that end, we’re working towards that goal fully in the open, and as I made reference to last week, this is month for it.
I spent time today filing a bunch of bugs on the various issues we need to fix to get this to work, and made them all dependent on the tracking bug for this effort.
We also have a sorta-Joelish Google spreadsheet to help track the effort.
If you’re interested in grabbing one of the bugs or generally helping out with the automation effort, please feel free to mosey into #build.
I can’t promise that it will be the most glamorous work in the project, but it’s some of the most-used and -useful work, and it will be appreciated—even if indirectly—by… well… millions of users.
In 40+ locales.
On three platforms.
______________________
1 The issues involved with releasing desktop software are not entirely similar to those related to releasing web-only software; that’s not to say one is easier/better or harder/worse than the other, but rather that a lot of release engineers from my “generation,” shall we say, have very limited desktop release experience, becuase they’ve only worked for dot-coms, and thus they were focused on website/web-app pushes.
2 As bsmedberg asked me in #build last week, “that’s making things unhappy on msys… do you remember why it was necessary (shouldn’t be)” and my answer was something along the lines of “No I don’t, but I remember it was breaking something.”
3 Read as “requiring human intervention, thus burning through people as if they were crumpled newsprint”
4 As in wall-clock time
5 Or, at least see how the magic happens.