You want a release? Well here’s your #@($! release!
I’d just like to state for the record that the 1.5.0.9/2.0.0.1 release was quite possibly the suckiest release I’ve ever worked on.
And if any of you remember the sleeping-in-the-office-for-a-week-1.5.0.2 release, that’s saying a lot.
It’s also saying a lot because the entire release went relatively smoothly1 until… well… today, when everything promptly started sucking. Hardcore.
Like most aviation accidents, it wasn’t a single thing that caused so much doom and strife. It wasn’t the air traffic controller who forgot to call out the traffic… or the maintenance guy who didn’t tighten all the bolts… or the pilot who got distracted looking at a faulty warning light.
It was all of those things, working in concert with each other.
Similarly, the list of what-the-hell-were-we-thinkings2,3 includes things that, on their own, wouldn’t have been a big deal at all. But compound them and you basically had fourteen hours straight of me banging my head against the wall and wanting to stab my face off.4
The sole thing that makes horrendous releases like today’s at least bearable is the community’s response and attitude when problems like this arise: the web team started working on the release in the morning, and remain responsive throughout the day; IT was around and helped debug deployment and database load and app cluster problems with us; the app team weren’t bothered a bit by being called in the middle of the night to look at weirdness; my build colleagues pitched in to help run some extra verification while I worked on deploying the contingency plan; and QA stayed until the bitter end (past midnight!) to make sure it all worked and to sign off on it all.
I don’t ever want to repeat a release day like this one.5
But—at the risk of sounding sappy—this pervasive and consistent attitude within the Mozilla project is what gives me the energy to face even the possibility of such beasts again.
_____________________________________
1 At least, no bumpier than any other Moz release I’ve ever worked.
2 Hindsight = 20/20, and all that…
3 My “WTF,” while I think defensible, is particularly… WTFular.
4 I must admit, Winnebago Man did help at about hour ten or so.
5 And I hope we’ll spend some time figuring looking at exactly what happened, so none of us have to.