She sure was a good binary format. Farewell, PPC… and we thank you.

06/21/2006

In the interest of reclaiming cycles to do more builds, I’d like to stop building nightly PPC builds for the 1.8 branch and trunk for Firefox, and possibly Thunderbird, but I have to talk to mscott first.
We’ve been doing both Universal builds and PPC builds for awhile now, and I’m pretty sure that everyone is wanting Universal builds over PPC builds.
Please feel free to hollar at me if this is not the case.

“First rule in government spending: why build one when you can have two at twice the price?”

05/18/2006

rhelmer applied the first release tag to the tree—FIREFOX_1_5_0_4_RC3—since the CVS server upgrade.
It took just under four minutes to complete.
For comparison, the old CVS server?
Forty minutes. On a good day morning at 4 am.
(This is the point in the story where we all go hug justdave.)

***
Went to the Flock presentation this morning on the Browser Technology track. I’ve never used Flock, so it was interesting to see the demo and look at some of the features they have.
I asked what their automated update story was, since there have been two or three releases of Firefox since the release they’re using (which, as I understand it, is 1.5 still).
They said “We couldn’t find the code for the automatic updates stuff.” Which we know about and are working on fixing. (In fact, I’ve been working on it at XTech!)
So, how did they solve this problem?
“We wrote our own replacement.”

Microsummaries + performance data = a Sheriff’s best deputy?

05/17/2006

I found Myk‘s XTech talk on microsummaries very interesting.
Last night at dinner, I believe it was Axel who was suggesting that publishing tinderbox performance data as an RSS feed might offer some interesting possibilities.
At first, I didn’t see the point of doing that exactly, but with microsummaries, a tree sheriff could put all the branches they’re supposed to be watching in their toolbar, so they wouldn’t have to scan a huge tinderbox page all day. Could maybe even whip up some XSLT (was it?) to make them change colors if the performance numbers jump outside of some pre-defined range.
Of course, the cool thing about microsummaries is you don’t necessarily need the RSS feed, it sounds like.
It’s an interesting idea, though… is there a more consumable format for perf data than we currently offer/publish?

Jitter Bug

05/17/2006

One of the largest hurdles with the virtualization migration plan was this huge unknown question of whether or not the tinderboxen performing tests could be virtualized.
Now that we have one (somewhat modern) tinderboxen—argo— cloned in a VM and running in physical hardware, we do have some data to look at.
“argo‘s” data is actually a bit confusing, because while the machine instance was cloned, not migrated, the machine’s identity was cloned; that is, “argo” on May 10th was a physical machine; “argo” after May 10th was a virtual machine. And then, “argo” once again became a physical machine on May 16th, with the virtual machine copy appearing on the tinderbox page as “argo-vm.”
On to the graphs!

Read More

Ahead of the Release Curve III: Virtually There

05/06/2006

The How-Does-This-Affect-Me? Version
Various Tinderboxen will be down next week, in cycles, so we can migrate them into virtual machines.
These migrations will start on Tuesday, 9 May, and will be performed in three rounds, with about four tinderboxen per round.
During each round, these machines will be unavailable for a 24 hour period. The migrations will not affect the Bon Echo Alpha 2 release plans.
The Short Version
Step 1. Move all Tinderboxen to VMs
Step 2. ????
Step 3. PROFIT!
The Long(er) Version
Starting with planning and help from Chase, over the past five months, we’ve been working towards migrating all of Mozilla’s Tinderboxen into virtual machines.
For those not familiar with the technology—VMware and Xen are players in the space with a lot of name recognition; Microsoft has an offering too, but it makes me giggle—virtualization offers the ability to run multiple instances of a full-blown operating system and an associated work load on the same piece of hardware. These OS instances are isolated from each other (conceptually, at least).
We’ve already migrated certain branches to virtualized Tinderboxen. Currently, the Firefox and Thunderbird maintenance branches are built using virtual machines. The 1.5.0.1, 1.5.0.2, and 1.5.0.3 releases have all come from VMs.

Ahead of the Release Curve II: The Disappearing Act

04/20/2006

One of the surprisingly common reports we get in #build are missing build reports: “The builds in latest-foobranch are five days old! Double-ewe-tee-eff!?”
Like a partner in a dysfunctional marriage, Tinderbox is an enabler of this bad behavior because after a certain amount of time, it just drops builds that haven’t reported in without letting anyone¹ know, so often times, by the time we receive a complaint about missing builds, they’re not a day or two old, they’re five or six days old.
This is unacceptable, and the Mozilla Community deserves better.
Having said that, with everything that the release team is typically doing during any given cycle, we don’t have the bandwidth to sit there and monitor tinderboxen to make sure that every single one is building what it’s supposed to. Often, this list changes so quickly that the documetnation about what each is supposed to be building isn’t even correct.
Rob and Dave have been working on fixing this, though, with—drumroll, please—automation.
Now, Nagios monitors the contents of ftp.m.o, and we get an email whenever builds in the latest-* directories for relevant branches are more than a day old. And we continue to get this email every few hours until it gets fixed.
This should help to cut down on having to let us know that builds haven’t shown up for five (or even more) days. It’s always been a reporting problem, as we’ve been typically able to respond to Tinderbox machine issues within 24 hours.
The lesson here is twofold:

• Automation will save us all² And we’re working on deploying more of it.
• Spam really is the best motivator to get stuff fixed

The next time you notice that the nightly build you were expecting to exist actually does… be sure to think of Rob and Dave.
___________________
¹ Who can do anything about it… yes, these abondonings are announced in IRC, but in ways that seldom get noticed.
² This, of course, isn’t anything new… but it’s nice to be in a place where we have the bandwidth to really start working on automation projects^3
3 And we’re working on even more projects I haven’t had time to blah-g about…