Last time I did 2.0 release candidate localization builds, I ran into a problem with the tagging and checkout.
For various reasons, we only tag the locales that we ship for a particular release with the _RELEASE tag. During the build, client.mk checks out every locale specified in all-locales. This has worked beautifully for the 1.5.0.x maintenance series, since the locales we ship aren’t shifting a lot, there aren’t many, if any, new locales on that branch, and if you check out a directory that isn’t tagged, you get… nothing. Exactly what we want, right?
Well, we repeated this process for the 2.0 RCs, and suddenly, the builds started failing with “cvs [checkout aborted]: no such tag ". This was certainly a surprise; “I had just created the tag,” I kept thinking to myself.
“Am I going insane?”
I started debugging it, and was only able to reproduce it once originally. Once I got a checkout going, it seemed to work repeatably, and since I was busy with 2.0 RC 2, I didn’t investigate more.
Well, it happened again with today’s l10n builds. Originally, rhelmer and I thought it may have been a problem of using the wrong CVSROOT, since cvs-mirror.m.o only gets updated every few minutes, and I had just created the tag. This didn’t make a huge amount of sense, as the command is run with -d, to specify the CVSROOT directly. A peek at the source confirms that -d takes precedence over $CVSROOT and CVS/Root. Then I thought maybe it was a compatibility problem. It turns out that we use CVS 1.11.2 on the client side to create release tags; maybe this is so old—it was released in 2003—that it was hiccuping with something server side?
After trying to reproduce this problem for rhelmer, I was only able to reproduce it once before it worked. Again. Something must be modifying the state server-side.
After some experimentation and more source reading, it turns out that an “optimization” introduced in the CVS 1.11 line, so-called “val-tags”, is responsible for the bug.
In a nutshell, when using val-tags, CVS searches for the existence of a tag by 1) looking into val-tags, and then b) looking at the RCS files themselves. This normally isn’t a problem, except in the case where an untagged directory is requested before a tagged directory. In the case of RC2, l10n/af was not part of the release, and therefore untagged, but l10n/ar was tagged. They were checked out in that [alphabetical] order.
Running “cvs co l10n/af l10n/ar” will repeatably produce the (incorrect) “invalid tag” error until you run “cvs co l10n/ar” (or some other checkout for which the tag does exist first. This adds the tag to the “val-tags” file, and after that point, CVS will check the val-tags file first, to see the that tag does indeed exist, and then traverse all of the directories you’ve listed, instead of the first one.
All the gory, buggy details are in tag.c‘s tag_check_valid(), which, based upon my very cursory reading of the source code, still exists in CVS 1.11.22.
While reading through the source, I was surprised at the number of comments that… didn’t inspire confidence:

/* FIXME: This routine doesn't seem to do any locking whatsoever
(and it is called from places which don't have locks in place).
If two processes try to write val-tags at the same time, it would
seem like we are in trouble.  */
/* FIXME: should check errors somehow (add dbm_error to myndbm.c?).*/

But, unlike so many other open source projects, at least we have…

/* warm fuzzies */
if (!really_quiet)

I think the moral of today’s story is: open source is cool because you can look at the source… but if you ever do, there’s a very real chance you could become very depressed.
Or scared.
Or both^1,2.

pacifica-vm, which most of you probably know as the Firefox 2 Windows nightly build machine, has had an interesting week.
Last week, I took the machine down to back up its virtual disk image, increase the amount of RAM available to it, and, in an attempt to decrease the cycle-time, added a VMware virtual-CPU to the VM, increasing it to two.
This didn’t really have the intended effect. Cycle times for both nightly and depend builds went up by about 15%.
Thinking that maybe the build system wasn’t making as efficient usage of its shiny new virtual CPU as it could, I upped make’s -j value from 3 to 4. This reduced cycle times… to what they were before the memory/CPU “upgrade,” but also had the useful side-effect that make would hang every few builds, including most notably on nightly builds. (That is, incidentally, why nightly builds on Tuesday, Wednesday, and Thursday were all late; make kept hanging overnight with -j4.)
Finally, last night, I removed the second VCPU, but kept the extra memory and higher -j values.
That change not only made the machine start reliably producing nightlies again (or, at least, make stopped hanging), but it took the cycle time down to 40 minutes for a depend cycle, and 2ish hours for a nightly build. (Interestingly enough, that full build value seems to fluctuate anywhere from about 90 minutes to just over a couple of hours; I think this is because the trunk build machine and the 1.8 build machine are on the same VM box, and they’re both starting their nightlies at the same time, which slows both of them down a bit.)
So, to recap here:

• Linking Firefox, especially on Win32, takes memory. A lot of memory. In the couple of trials I paid attention to, it took around 700 megs. Seeing as the VM had 700 megs, a large part of the problem seemed to be the machine descending into swapping thrash-hell when trying to do the final link.
• Win32 SMP = Teh suck. I had actually learned this from previous experiences in previous lives, but… a reminder is always good.
• When you actually pay attention to VMs, and spend some time “tuning” them—which in this case, amounted to creating a better match between the memory profile for the machine’s task and the virtual hardware—VMs don’t perform all that badly, relatively to physical hardware. gaius-vm, for instance, has horrible cycle times compared to gaius, but it’s not “just because it’s a -vm.” It’s because no one’s paid enough attention to it after migrating it to tune it. (No, I haven’t taken what I’ve learned here and applied to gaius.)
• Once again, sometimes… VMware['s performance] continues to confuse the hell out of me.