A First Timer, All Over Again
[Editor's note: this was written on Thursday evening, but I spent my weekend moving1; by the time I got Internet Tubes installed at the new place, the weekend was over.]
Astute readers may have noticed we released Songbird 0.5 last night. It’s got some pretty cool fu in it, too! Definitely worth playing with.
A lot of people worked really hard on this release, putting in plenty long hours and a couple of long weekends.
The first release any release engineer does in a new environment is noteworthy. It usually feels scarier than it really is, and you end up taking a lot of notes, not only for the “gotchas,” but for the “Oh man, we gotta fix that”-s.
It’s weird that I vividly remember the first product I shipped2 ever since I’ve been a build engineer.
At VMware, it was a pretty disconnected, formal process involving creating tarballs of specific formats for another group3 to get them ready for the website. There was some weird rule about only being able to do so many releases at certain points in time, because the website could only handle so much traffic. There was also an ISO image that got created for those Enterprising individuals who only took their software in bits smushed onto plastic-form.
It was all very interesting, but being an enterprise product, the team seldom got any feedback, good or bad, until a couple of months later when “enough early adopters” had installed the new software to get an idea of what was and wasn’t broken. In some sense, it was4 something like sending your kid off to college… knowing you’d see them again for Christmas, but that’s about it.
Contrast that with my first Mozilla release, where the Community was abuzz with news of the release5, and the second I posted the bits, people started downloading them. I vividly remember sitting at my desk, ready to press Enter on the rsync command that would dump updates out to millions of users.
Chase was standing behind me. “Looks good,” he said, and patted me on the back. I rolled my chair back from my keyboard, and stuck my finger out to press the key. As my finger got close to the keyboard, I cringed, not unlike someone detonating a suitcase bomb, and not knowing if there were firecrackers in it, or C4.
When it was all said and done, it didn’t matter which it was; I had done it right6. Chase chuckled as he went back to sit at his desk. I think he knew it at the time, but pressing “return” got progressively easier with each subsequent release, even though it was updating more and more people.
At Songbird, the experience was much the same, and given that the technologies involved are similar, it was mostly an exercise in learning where Songbird’s engineers had moved the mines in the field that is Mozilla’s Release Process.7
Songbird’s requirements, of course, are very different than Firefox and Thunderbird’s, and it’s been very interesting to see how the same problem was addressed with different contextual constraints.
The handoff mechanics are similar to Firefox: updates get verified in a staging environment, installers then get posted, and then updates get pushed to production8. I had to learn to shut up when Redfive was explaining something to me, since lots of things really are the same in the Birdnest, but lots of things are not, and the reasons behind those deltas are really important. I didn’t even notice that I was doing the same bomb-in-suitcase maneuver until Redfive asked me about it as I was getting ready to run the push_production_all script.
Speaking of history repeating itself, we had a minor problem with AUS updates that turned out to be due to a typo on my part9. In a weird way, I felt right at home again. (It probably had something to do with having to read updater’s source code.)
I must admit, the one thing I love about both release processes like Songbird and Mozilla’s is the fact that you run a command, and Stuff Happens ™.
Immediately.
People get the software, and they’re using it, and they want to tell you about that experience immediately. It’s much more like sending a kid off to kindergarten: you know you’ll get to hear about their day shortly, and it’s almost always either a really good day, or a really, really bad day10. Either way, it keeps you engaged, and you can work on helping make the next time around better.
Songbird does have one thing up on Mozilla’s release process: right after we shipped and updates starting permeating the ‘Net, we all gathered around to savor a bottle of scotch.
Any release checklist that has that codified on it11 is definitely a place I can get on board with.
__________________________
1 More on that shortly…
2 Which is, of course, overloaded in our industry, but doubly so in ReleaseLand, since “shipping” usually involves being the person to see the product out the door
3 IT, as I remember
4 I can only imagine…
5 As they always are
6 Thanks to QA, as always, for helping me make sure I did get it right
7 Which, having implemented portions of that process, in policy and code, I say with an unwavering amount of love
8 All while other teams are busily working on things like release notes and website updates that I’m, frankly, quite happy not to have to work on
9 Yah, we’ve NEVER HEARD OF THAT BEFORE.
10 But don’t read too much into that analogy, though.
11 The closer the drinking is to the end of the checklist, of course, the better; this metric actually turns out to be a pretty good measure of the quality of an organization’s release process…