The Dark Ages That Never Were


I generally don’t discuss previous work experiences with the details unredacted—”Names changed to protect the innocent and/or guilty,” and all that—but Mozilla prides itself on its openness and transparency1, and they’ve been asking for #webstories, so here’s mine:

Preed & Moz

A very young, very naive me, in the lobby of Netscape’s Building 23.

My beginnings with Mozilla and, perhaps ironically, with release engineering started when I was 18. I worked for a summer on Netscape’s Build/Release Engineering team, and fell in love with the product, the culture, and release engineering itself.

To say that it shaped my career is not an understatement.

When I went off to college, I remained a part of Mozilla’s burgeoning community, working on various aspects of the project. So when I joined Mozilla Corporation in 2005, it felt a lot like returning home.

In many ways, it was the wild wild west at that time; I did my best to do the core work of any release engineer—shipping the damn bits—while balancing that with forward-looking initiatives. For a period, I was the only full-time release engineer.

Mozilla Corporation’s current Director of Release Engineering, John O’Duinn, has been presenting Release Engineering as Force Multiplier2.”

In it, O’Duinn makes some bold claims and based on reactions, audiences seem pretty horrified about Mozilla’s release engineering past.

Release Engineering in those days, like any team in any organization, wasn’t perfect by any stretch of the imagination; it was a role that had been treated a bit like hot potato, handed around from engineer to engineer since the Netscape days, and it is accurate to say it hadn’t really been invested in.

But O’Duinn does not need to make false statements to make this point.

  • O’Duinn claims that “chemspill releases”3 took 4-6 weeks to ship.

    I don’t know where that number comes from, but we shipped a handful of “chemspill releases”4 in at least the 11 hour time frame that are today’s reported numbers.

    I know this, because I was there… shipping the releases he’s referring to5.

    Many “chemspill releases” did take longer than 11 hours6, but nowhere near the 4-6 weeks O’Duinn cites.

    (Maybe he’s conflating the standard security/stability releases of that era, which did have rolling 4-6 week release schedules, but that is not what the slide says and it’s not the impression O’Duinn’s explanation gives. It never took 4-6 weeks to ship “chemspill releases” once Engineering gave its “GO” message, as he puts it.)

  • O’Duinn recounts an anecdote about having to tell executives that Mozilla Corporation had to stop supporting partner builds, a revenue generator:

    “This was so intrusive to the little bit of human-time that was available in Release Engineering that we had to actually stop doing this in 2007. Nothing upsets a CEO more, when he’s trying to raise revenue, than having to turn away customers with cash in hand.”

    These statements are factually true, but they omit a hugely relevant contextual detail: Mozilla wasn’t set up (either organizationally or technologically) to support the kinds of partners that were knocking on the door.

    These partners7 expected a certain level of direct access and support that the Mozilla project had, to that point, not been too familiar with8. An entire support team was created to handle these requests and coordinate work with other teams, such as marketing and legal teams, which these sorts of interactions required.

    Additionally, there were some technical rough edges, especially around vendor-supported application updates; these needed to be resolved to support these builds at scale and Engineering and QA bandwidth for that effort had to be found.

    To intimate that Release Engineering was the cause for “turning customers away” is disingenuous at best.

More time could be spent pointing out context-lacking or misleading-statements9, but these two were the most egregiously incorrect, in my opinion.

I’m proud of my work at Mozilla.

Under the management structure at the time and with the resources I was afforded, our small team accomplished a lot. I’m proud that I was able to serve a growing worldwide community, and not once introduce a user-visible issue, due to a release engineering error.

I’m not necessarily proud of the mechanics of how that was accomplished on occasion, but if you look at the outcomes, I think what I, and the other release engineers before me did to support Mozilla and Firefox in those early days was nothing short of amazing.

O’Duinn’s talk largely omits this history, and where it doesn’t, he paints an opaque, context-less tapestry that make it sound as if shipping a Mozilla release was something out of the Dark Ages.

Similarly, I am proud of what Mozilla, and the release engineering team I left, have become since.

They employ a lot of interesting techniques and have responded well to the market’s demands. Because they ship software consumers still have to install natively, theirs is a realm which presents many unique problems and they’ve invented and implemented many interesting, noteworthy solutions around that problem-space.

But while it may not have been elegant, Mozilla—the community and the Corporation—did ship software before May, 2007.

Using vague statements and false claims to paint a picture of a Dark Age that never was, not only denigrates the hard work of the engineers on the Netscape (and later Mozilla) Release Engineering teams before O’Duinn’s tenure, it clouds the credibility of what is otherwise an interesting, useful, and important message: that release engineering really is important to the business of shipping software.

1 Specifically “Transparent community-based processes promote participation, accountability, and trust.”
2 I like the idea, but I could do without the industry’s increasing use of military metaphors
3 Those involving security-related or Web-breaking fixes
4 We didn’t call them that back then
5 My friends and family can recount the many occasions I canceled plans with them to spend a night or weekends shipping those releases, with sometimes pretty negative impacts on those relationships.
6 Usually on the order of one to three days
7 Many of them large corporations themselves
8 There were the concept of partner builds, but they were largely repacked Firefox builds that had been sold to companies and supported by independent consultants
9 And there certainly are plenty to discuss