The Dark Ages That Never Were
I generally don’t discuss previous work experiences with the details unredacted—”Names changed to protect the innocent and/or guilty,” and all that—but Mozilla prides itself on its openness and transparency1, and they’ve been asking for #webstories, so here’s mine:
A very young, very naive me, in the lobby of Netscape’s Building 23.
My beginnings with Mozilla and, perhaps ironically, with release engineering started when I was 18. I worked for a summer on Netscape’s Build/Release Engineering team, and fell in love with the product, the culture, and release engineering itself.
To say that it shaped my career is not an understatement.
When I went off to college, I remained a part of Mozilla’s burgeoning community, working on various aspects of the project. So when I joined Mozilla Corporation in 2005, it felt a lot like returning home.
In many ways, it was the wild wild west at that time; I did my best to do the core work of any release engineer—shipping the damn bits—while balancing that with forward-looking initiatives. For a period, I was the only full-time release engineer.
Mozilla Corporation’s current Director of Release Engineering, John O’Duinn, has been presenting Release Engineering as Force Multiplier2.”
In it, O’Duinn makes some bold claims and based on reactions, audiences seem pretty horrified about Mozilla’s release engineering past.
Release Engineering in those days, like any team in any organization, wasn’t perfect by any stretch of the imagination; it was a role that had been treated a bit like hot potato, handed around from engineer to engineer since the Netscape days, and it is accurate to say it hadn’t really been invested in.
But O’Duinn does not need to make false statements to make this point.
- O’Duinn claims that “chemspill releases”3 took 4-6 weeks to ship.
I don’t know where that number comes from, but we shipped a handful of “chemspill releases”4 in at least the 11 hour time frame that are today’s reported numbers.
I know this, because I was there… shipping the releases he’s referring to5.
Many “chemspill releases” did take longer than 11 hours6, but nowhere near the 4-6 weeks O’Duinn cites.
(Maybe he’s conflating the standard security/stability releases of that era, which did have rolling 4-6 week release schedules, but that is not what the slide says and it’s not the impression O’Duinn’s explanation gives. It never took 4-6 weeks to ship “chemspill releases” once Engineering gave its “GO” message, as he puts it.)
- O’Duinn recounts an anecdote about having to tell executives that Mozilla Corporation had to stop supporting partner builds, a revenue generator:
“This was so intrusive to the little bit of human-time that was available in Release Engineering that we had to actually stop doing this in 2007. Nothing upsets a CEO more, when he’s trying to raise revenue, than having to turn away customers with cash in hand.”
These statements are factually true, but they omit a hugely relevant contextual detail: Mozilla wasn’t set up (either organizationally or technologically) to support the kinds of partners that were knocking on the door.
These partners7 expected a certain level of direct access and support that the Mozilla project had, to that point, not been too familiar with8. An entire support team was created to handle these requests and coordinate work with other teams, such as marketing and legal teams, which these sorts of interactions required.
Additionally, there were some technical rough edges, especially around vendor-supported application updates; these needed to be resolved to support these builds at scale and Engineering and QA bandwidth for that effort had to be found.
To intimate that Release Engineering was the cause for “turning customers away” is disingenuous at best.
More time could be spent pointing out context-lacking or misleading-statements9, but these two were the most egregiously incorrect, in my opinion.
I’m proud of my work at Mozilla.
Under the management structure at the time and with the resources I was afforded, our small team accomplished a lot. I’m proud that I was able to serve a growing worldwide community, and not once introduce a user-visible issue, due to a release engineering error.
I’m not necessarily proud of the mechanics of how that was accomplished on occasion, but if you look at the outcomes, I think what I, and the other release engineers before me did to support Mozilla and Firefox in those early days was nothing short of amazing.
O’Duinn’s talk largely omits this history, and where it doesn’t, he paints an opaque, context-less tapestry that make it sound as if shipping a Mozilla release was something out of the Dark Ages.
Similarly, I am proud of what Mozilla, and the release engineering team I left, have become since.
They employ a lot of interesting techniques and have responded well to the market’s demands. Because they ship software consumers still have to install natively, theirs is a realm which presents many unique problems and they’ve invented and implemented many interesting, noteworthy solutions around that problem-space.
But while it may not have been elegant, Mozilla—the community and the Corporation—did ship software before May, 2007.
Using vague statements and false claims to paint a picture of a Dark Age that never was, not only denigrates the hard work of the engineers on the Netscape (and later Mozilla) Release Engineering teams before O’Duinn’s tenure, it clouds the credibility of what is otherwise an interesting, useful, and important message: that release engineering really is important to the business of shipping software.
_______________
1 Specifically “Transparent community-based processes promote participation, accountability, and trust.”↵
2 I like the idea, but I could do without the industry’s increasing use of military metaphors↵
3 Those involving security-related or Web-breaking fixes↵
4 We didn’t call them that back then↵
5 My friends and family can recount the many occasions I canceled plans with them to spend a night or weekends shipping those releases, with sometimes pretty negative impacts on those relationships.↵
6 Usually on the order of one to three days↵
7 Many of them large corporations themselves↵
8 There were the concept of partner builds, but they were largely repacked Firefox builds that had been sold to companies and supported by independent consultants↵
9 And there certainly are plenty to discuss↵
I was reading something along these lines the other day. Not so long ago, it was routine for people to fudge their stories to match the message they’re trying to send, but you can’t do that anymore. The people who know better will find out and speak up.
You helped create this world with that little browser thingy that you shipped.
Disclaimer: I was a part-time release engineer at Mozilla when preed joined, although most of my time was taken up with QA tools. I was there when joduinn joined, when preed left, and am now a release engineering manager at Mozilla under joduinn. It should go without saying, but I’m speaking for myself.
There’s a bunch of stuff to unpack here.
First, let’s start with the human bits, because they are pretty important and colour everything else.
There is no love loss between you and joduinn, that much is clear. I have a good, working relationship with joduinn, so my interpretation is necessarily more charitable.
As someone who cut-n-pasted the release commands from the wiki for early Thunderbird 1.5 dot-releases, I don’t see joduinn’s presentation as condemning your work to the Dark Ages. Leaf, Chase, yourself, and rhelmer did what needed to be done with the tools at hand to “keep the lights on,” as we phrase it on the goals sheet these days. Without that work, there *is* no Mozilla today. joduinn is talking more about the transition that release engineering needed to make as the competitive landscape (and hence, Mozilla needs) changed.
I think that many of the points you have here are valid. However, I’m not sure why you didn’t bring up them up at the conference. The contextual detail you’re striving to add is important, but the impact is lost by being delivered after the fact. I hope that the issues between you and joduinn didn’t stop this discussion from happening at the conference, because it might have led to a more nuanced conversation by everyone.
Some specific things in the post I’ll call out:
That’s not the reaction I saw, or at least not the one that my random conversational sampling in the afternoon revealed. A lot of release engineers I talked to had an involuntary shudder because they are either suffering through something similar now, or are repressing those memories from previous jobs.
I haven’t done the actual math, because…well, math, but there’s another part of joduinn’s presentation that provides context.
joduinn also talked about how the perception in the rest of the company was that releases took forever, and that releng was the bottleneck. More careful inspection of the process revealed lots of other speedbumps (and tiger traps) unrelated to releng within the release pipeline.
It’s quite possible that actual releng time for some of those early release was low, but that data was lost in a sea of missed hand-offs between QA, webdev, etc.. That lack of unifying process *across groups* ballooned the end-to-end release time far beyond what was required to “simply” build the bits, push to mirrors, and update bouncer.
I believe that’s the point: if we were technically unable to repack builds for partners, no amount of extra process around partners makes that work.
FWIW, we’ve solved the technical bit now, but you’re absolutely right about the amount of extra effort that partner support requires. Mozilla is still struggling to work at scale with partners.
I think this is exactly joduinn’s point: we have a process now where release engineers don’t need to work themselves to death, and I don’t need to watch my friends and co-workers get burnt out and leave the company.
Hey Coop,
Of all the people that could respond, you’re one of the most equipped to do so, so thanks for chiming in.
With all due respect: don’t put words in my mouth. (Or John’s for that matter.)
My opinion regarding John (not that you and I have discussed it recently) is irrelevant to the issue at hand, which is: O’Duinn making (and repeating) false statements in his presentation.
I don’t agree that the impact is lost, nor am I sure why the venue I raise the issue in is relevant to the discussion. It seems like a distraction.
Having said that, I was busy at the conference, covering it for The Ship Show, and wanted to let his presentation percolate in my head before responding. It was also helpful to review the video from his tech talk he gave a week later, the content of which was consistent with my memory of the workshop keynote.
Our interpretations may differ (“horrified” vs. “involuntary shudder”), but it seems like we both observed… some sort of negative reaction, at least.
It often was lower, and yes, coordination was sometimes an issue.
Having said that, the picture O’Duinn paints omits the context about the fact that we were shipping bits with people usually in the same room (and those that weren’t we very tuned in over IRC); the costs of doing a respin were high, but not as high as O’Duinn portrays (how he could claim he knew what they were is confusing to me; I don’t know where he’d get that data), precisely because the team was smaller then.
I agree that the perception across the company may have been different, and that’s a great point to bring up.
So… we agree on this?
In his tech talk, O’Duinn said, and I quote: “This was so intrusive to the little bit of human time that was available in release engineering that we actually had to stop doing this in 2007. Nothing upsets a CEO more when he’s trying to raise revenue than having to turn away customers with cash in hand.”
My point was that solely blaming Release Engineering for the inability to take on [more] partners is disingenuous; it was a organizational-wide problem that needed to be addressed holistically, not [my own paraphrasing] “This one department screwed the company before we hired a Director to fix it.”
That’s a good development to see; but it’s not the point I was making. I was clarifying that I (and all of us back then, really) pulled some pretty extreme hours, precisely to ensure shipping “chemspill” releases was as fast as possible, and significantly less than the 4-6 weeks O’Duinn claims.
In any event, it looks like we agree on the core points I made in my original post, which is nice to know, especially coming from someone who was in the tranches in that era with me, and who lived it himself.
(I note for the record that preed attended in person the RelEngConf where I presented this talk, and sat front-row-center for the duration, including the Q+A session afterwards. Preed+I talked in person a few times during the rest of the day’s proceedings, and also by email several times in the week after RelEngConf. As we used to work together in Mozilla, preed also has my email address and (likely) my cellphone info which hasnt changed. At no point did preed give any indication of his concerns, nor contact me with any questions about my presentation, prior to publishing this blogpost.)
hi preed:
1) Your claim about having shipped a release in 11hours surprises me. Even just the minimum compute time needed for windows builds followed by l10n repacks followed by generating the updates used to take way more then 11 hours – and thats not counting other time-consuming steps in a release, like signing-of-builds, mirror-caching, WebDev and actual QA!
To make sure we are talking apples-to-apples comparison here, when I say “shipped a release”, I am talking about all the cross-group work done in the time from “go to build” to “users can see the release (either by downloading a new installer installer or getting updates)”. Each release used to take multiple weeks. A respin/rebuild of a release could take the same amount of time, depending on how far into the release RelEng were asked to abort-and-restart. Muti-product chemspills (or firedrill releases as we called them back then) were handled by the simplistic tactic of doing one chemspill at a time, focusing first on the release with the most users, and once that release shipped, then focusing on the next most popular release. Sometimes we could be lucky and have release engineers freed up to start the second release before the first was fully completed. Sometimes we could be unlucky and have to respin/rebuild the entire set of releases because of late-breaking problems found by QA. This unpredictability was bad for the humans and for the organization. Regardless, the clock started with the “go” when the code was ready, not when RelEng was ready to start the build(s).
After digging through my incomplete personal email archives and blogs for the last 6 years, the major milestones to me were our first release within a week (FF2.0.0.7 http://oduinn.com/blog/2007/09/21/firefox-2007-by-the-wall-clock-numbers/ ), our first release within 24 hours (FF3.6.6 http://oduinn.com/blog/2010/06/29/firefox-3-6-6-by-the-wall-clock-numbers/) and our 8-releases-in-48hours chemspill in Feb 2013. Each being testimony to the ongoing improvements in our automation.
2) Partner builds: During my interviews at Mozilla, I was told by executives that they had already decided to stop accepting new contracts for partner builds because of infrastructure problems… and they asked how would I go about fixing this if I was hired. Outside of RelEng, Mozilla already had a small, but sufficient, staff of people hired for the purpose of establishing and supporting partner agreements. However, it was more important to the entire company that the overloaded RelEng group focus on fixing generic infrastructure and release problems first before fixing infrastructure for partner builds. Therefore, those partner teams had been told to stop accepting new partner contracts and were instead refocused onto other work until Mozilla’s RelEng infrastructure was able to safely support the mechanics of the existing partner repack contractual obligations. This was a triage decision I understood.
3) Netscape+Mozilla obviously did ship software prior to May2007, and while the process wasn’t perfect, I think the term “Dark Ages” is uncalled for. In my presentation, I noted that prior to my starting in May2007, Mozilla shipped all platforms and all locales at the same time and on the same changeset. This is still true today, and is still better then most of the software industry – a point I noted in my presentation as being of great value to Mozilla, to our users and to be encouraged across the industry. Yes, I did also note there was lots of room for improvement. Given the tooling at the time, each release shipped at great personal cost to the human release engineer of the day. I personally slept under my desk at Mozilla during some early FF1.5.x and FF2 releases, like I know you, and other Mozilla release engineers before you, did also. I think we can both agree the process was unsustainable, and the staff turnover numbers for RelEng spoke volumes.
Summarizing 6 years of work into a 45minute presentation allowed people to see the before-and-after improvements side-by-side, and better understand the context of the tactics used to make it all possible. The improvements are impressive, and speak for themselves. In my opinion, the delivery pipeline here at Mozilla is now one of the best in the entire industry. We here at Mozilla are rightly proud of how this delivery pipeline allows us strategic possibilities in the marketplace Mozilla would otherwise not have, and does so without needing to continue sacrificing human release engineers like you experienced during your tenure here, and others did before you. Of course, there’s still more improvements we can do, and we in Mozilla’s RelEng keep making baby steps forward every day.
John.
Hey John,
Yes, I attended the conference.
Both you and Coop raised this point; I’m at a loss as to why it’s relevant. Maybe you could explain?
(Since you brought it up: it’s worth noting that we did have an email thread going after the conference; it was not related to anything raised in this post; and the thread ended when you did not respond to my email of May 22nd, despite there being loose ends to the question we were trying to resolve. You just dropped the thread entirely.)
Using your definition of “‘Go’ from dev” to “in users’ hands,” we shipped some beta releases in about that length of time; I actually went back and looked at a spreadsheet we generated for “the business” at the time, and the numbers aren’t 11 hours (they’re about triple that, which wasn’t bad back in those days), so I will gladly retract this statement, based on the way you’ve defined it.
Having said that, that quick perusal of my notes revealed a number of releases that we did get shipped in significantly less than the 4-6 week timeframe you keep repeating.
Firefox 1.5.0.6 is one such example.
2.0.0.15 is another. There are others, and a brief perusal of the Firefox release history page shows this (i.e. releases that had less than 4 weeks between them, would, by definition, refute your claim, since as you point out, we single tracked releases back then.)
By the way, 2.0.0.7 was not the first release done within a week; 1.5.0.6 (at least) was.
Also, it’s worth noting: your presentation makes it sound as if the priority for the “chemspill” releases you describe was getting the release out the door; but the releases you cite that took 4-6 weeks to ship included scheduled beta releases and periods and other intentioned delays, for various reasons. I don’t know why you purposefully confuse the audience over “Get this company-affecting emergency release out right now”-situation vs. “A scheduled security/stability release, which we planned on a 4-6 week cadence,” which was the norm back then.
We did collect some metrics on releases back in that period, but they seem to be on the Mozilla Corporation intranet. It should be possible to publish those old metrics, no?
The Corporation started publishing those documents publicly around the Firefox 2.0.0.7 timeframe. (Is that the earliest release with these documents made public?) That’d help provide a much more complete picture.
I stand by my statement that at that point in time, there were technology (i.e. Engineering) and support (i.e. Partner Support) constraints, in addition to Release Engineering ones. Coop’s comment seems to agree with my assessment. So, your claim that the staff was “sufficient” does not seem to agree with what others in trenches at the time experienced.
Great!
I’d love to see a version of this presentation that doesn’t paint the history that way (as indicated by numerous people observing your audience “involuntarily shudder[ing].”)
Your claim about sleeping under your desk during some early Firefox 1.5.x releases surprises me, given that 1.5.0.12 shipped a couple weeks after you were hired and was the EOL release. (My notes show rhelmer and Nick shouldered that release.)
I think the story you tell about Mozilla’s release engineering journey is a compelling (and necessary!) one for the entire industry to hear.
But you should tell the story based on facts, instead of omitting historical context and playing fast and loose with the details in order to construct a story arc which sounds good and tests well with audiences, but ultimately is largely a fiction.
Incorporating an accurate history of the age before your hire-date would make it a more compelling story than it already is, not to mention give the respect due both the people and the Mozilla of that era.
tat es ihr nicht schlechthin nach, sie hatte wahnsinniges Verlangen danach.
Ich sitze da allein, schaut mich , ihren Kopf leicht zur Seite gedreht als wenn sie sagen Tu es nicht du Idiot!.
Desire nothing, Chafe http://www.return-address-labels.info/item/label/single/106220314907999518/boat-anchor-return-address-label-pink-white.html not at fate, nor at Nature’s changeless laws. But struggle only with the personal, the transitory, the evanescent and the perishable.
garcinia forskolin green coffee acai foods mangosteen hoodia gordonii psyllium chitosan guar gum lipozene glucomannan
We http://www.weddingbusinesscards.net/i/240887870995099992/blue_save_the_date_wedding_website_insert_double_sided_standard_business_cards_pack_of_100.htm will either find a way, or make one
biblical pre marriage counseling worksheets
To be fair, I don’t know a single undergrad at Berkeley who wasn’t rejected from They’re still a little
it comes within a gorgeous box seems awesome still it is a bit smaller additionally concerning my personal moms wrist however it appearances great simply want that it had been further
Die glauben doch, der springt, weil ein Frosch drin ist.
Durch den tiefen Ausschnitt ist es empfehlenswert, unter dem Mieder eine Trachten- oder Dirndlbluse zu tragen.
Er war Mensch – ich finde, das ist heute mehr denn je ein Lob, wenn bestätigt bekommt Mensch zu sein, gewesen zu sein.
recycling dumpsters
this contact form http://www.shopez-moi.com/sitemap438.php
construction container rental
container rental near me http://www.jerboa-redcap-books.com/sitemap215.php
würden Liebesgeschichte wohl nie klappen :] Ach Moony, ich wollte Dir nur mal sagen, dass ich Dich total klasse finde…ich bin übrigens eine der Älteren Generation…war vor allem im alten Forum aktiv dabei.
Sonntäglich gewandet marschierte das letzte steile Stück, kaum achtend der eigentümlichen Landschaft, die meine Phantasie graue Frühzeit erinnert, als Pithek und Homo ängstlich schnatternd umherliefen, auf der Flucht vor falschen und echten Säbelzahnkatzen.
Kleingeld loswerden, Münzen wechseln Bank, Weltspartag Geld eintauschen, Geld einzahlen Sparkasse.
Die Zutaten, aus der meine neuste Live-Reportage auf Grossleinwand entstand: Vier Monate Zeit, ein 4×4-Pickup, ein Haufen Foto- und Videoausrüstung, eine fliegende Kamera – und kein Reiseplan.
Geil nicht wahr.
Die geile Tussi hat eine feuchte Muschi.
Besonders wenn sie den Zwillingen Sex anbieten.
Zum ersten Mal spürte ich nun ihren Fotzensaft auf meiner Zunge, etwas strenger als der Dagmars, aber ebenso aufgeilend.
Lass uns gemeinsam ordentlich ficken.
Pralle Titten fallen beinahe aus dem tief ausgeschnittenen Oberteil heraus, und der lange Rock hat einen beinahe ebenso langen Schlitz, der bis zur Hüfte hinaufreicht.
This goods had been with such a great price tag I not believed that excellent is so exceptional. It is breathtaking. This one mom does really like it concerning Christmas time day when she opens gifts and it appears like I invested far more, then again prices is really ideal!!
In der Kategorie Pornoflatrate ist Fundorado eines der beliebtesten Pornoseiten im Netz.
Sauna ist kein Leistungssport Für Anfänger bietet das Agrippabad einen Zwei-Stunden-Tarif .
I requested it concerning my mama not really expecting much as that was quite inexpensive. Once I obtained this I was thrilled! It’s beautiful & does not see affordable after all! The mother liked this!
I have the as being a present concerning my personal sister, happy mom. She liked this!! I was scared they had been going to be also small due to the earlier evaluations and exactly how they seemed when I took the top, but it in shape quite nicely. Things appearance striking upon plus their beautiful package it really is presented when.
Herzlichen Dank für die Spendenflut Farbflut und alle Spenderinnen und Spender!
Du hast nicht zuviel versprochen.
Maik sagte mir, dass dies sich auch nicht ungewöhnlich ist.