Friday, September 29th, 2006
pacifica-vm, which most of you probably know as the Firefox 2 Windows nightly build machine, has had an interesting week.
Last week, I took the machine down to back up its virtual disk image, increase the amount of RAM available to it, and, in an attempt to decrease the cycle-time, added a VMware virtual-CPU to the VM, increasing it to two.
This didn’t really have the intended effect. Cycle times for both nightly and depend builds went up by about 15%.
Thinking that maybe the build system wasn’t making as efficient usage of its shiny new virtual CPU as it could, I upped make’s -j value from 3 to 4. This reduced cycle times… to what they were before the memory/CPU “upgrade,” but also had the useful side-effect that make would hang every few builds, including most notably on nightly builds. (That is, incidentally, why nightly builds on Tuesday, Wednesday, and Thursday were all late; make kept hanging overnight with -j4.)
Finally, last night, I removed the second VCPU, but kept the extra memory and higher -j values.
That change not only made the machine start reliably producing nightlies again (or, at least, make stopped hanging), but it took the cycle time down to 40 minutes for a depend cycle, and 2ish hours for a nightly build. (Interestingly enough, that full build value seems to fluctuate anywhere from about 90 minutes to just over a couple of hours; I think this is because the trunk build machine and the 1.8 build machine are on the same VM box, and they’re both starting their nightlies at the same time, which slows both of them down a bit.)
So, to recap here:
| |
Nightly Build |
Depend Build |
| Before changes |
~ 2 hours |
~1 hour |
| After memory/CPU “upgrade” |
~ 2 hours, 20 minutes |
~ 1 hour, 15 minutes |
| After adding -j4 |
~2 hours; hung often |
1 hour; hung often
|
| Remove one VCPU |
~ 2 hours; jury still out, though |
40 minutes |
Things I’ve learned from this experience:
- Linking Firefox, especially on Win32, takes memory. A lot of memory. In the couple of trials I paid attention to, it took around 700 megs. Seeing as the VM had 700 megs, a large part of the problem seemed to be the machine descending into swapping thrash-hell when trying to do the final link.
- Win32 SMP = Teh suck. I had actually learned this from previous experiences in previous lives, but… a reminder is always good.
- When you actually pay attention to VMs, and spend some time “tuning” them—which in this case, amounted to creating a better match between the memory profile for the machine’s task and the virtual hardware—VMs don’t perform all that badly, relatively to physical hardware. gaius-vm, for instance, has horrible cycle times compared to gaius, but it’s not “just because it’s a -vm.” It’s because no one’s paid enough attention to it after migrating it to tune it. (No, I haven’t taken what I’ve learned here and applied to gaius.)
- Once again, sometimes… VMware['s performance] continues to confuse the hell out of me.
mozilla, planetmoz
Releng Machinery | 3 Comments »
V61
Thursday, September 28th, 2006
Mid said: “You should post the translated version of that Quote of the Day; it’s much funnier.”
So, I present… the TQOTD:
More seriously, if the spirit of Free, it is râler by saying that Firefox is not Libre because of the trade mark, I am not sure that that is a spirit which attracts me.
mozilla, planetmoz
blahblahblah | Comments Off
V60
Tuesday, September 26th, 2006
If you ever hear me complaining about crosswind landings, just remind me that I really don’t have all that bad…
***
I’ve been surprised—nay,
amazed—at how clear it’s been in the evenings here in the Bay Area lately. I’m hoping it stays that way all through winter, for it makes flying at night, one of my favorite times to fly, possible.
On a return from
KSBP on a recent flight, my pax got a
great picture of what it’s like to operate in a
clear night sky in the Bay.
***
I had threatened to take
morgamic up for a Bay Tour upon his next visit, so I just finished scheduling a plane for Friday afternoon.
I have one other seat available. Anyone else game?*
* Extra headset procurement, weather, and RC2 release schedules permitting.
mozilla, planetmoz
blahblahblah | 1 Comment »
V59
Wednesday, September 20th, 2006
I presented my overview of Mozilla’s Build System to Seneca’s Topics in Open Source course this morning:
13:33 <vlad> how'd preed's talk go?
13:34 <Moe> vlad: he had an awesome sport jersey
13:34 <Moe> with a lizard
13:34 <Moe> thats all i took from the lecture
13:34 <Moe> oh and of course there was that whole build thing
Glad to know I made a positive pedagogical impression.
***
Prepping for the presentation
was full of fun surprises.
I ended up mentioning this during the presentation because I think it’s important to point out to newcomers who may be working hard to integrate themselves into the community that even the more… seasoned among us still can have huge grey-to-black areas in our knowledge about parts of the project. That’s part of the project’s charm—its deep breadth of things to work on—
and its curse.
***
Afterward, I had lunch with
Dave Humphrey, the professor for the Topics on Open Source corse and
Chris Tyler, another professor in the department. We had a very enjoyable conversation about objectives in the Topics on Open Source course, ideas for material as it relates to other courses, and how the experience has been going thus far, for both student and professor.
Seneca, through this course and others they’re working on developing, is trying to bring more open source material into the classroom. This has a number of benefits for both students in the course and the open source projects they choose to work with.
The benefit to the body of open source code is obvious. The benefit for students is more subtle, a point I had spent some time trying to convince professors at my
alma mater of: quite simply, open source provides real world conditions that the classroom cannot.
When students go out into the workplace, the chances that their first job will entail designing and implementing software systems from scratch is very small. Rather, the “fresh meat” typically get all the bugs that no one else had time (or wanted to) fix. Most software jobs out of the gate are maintenance jobs.
Contributing, as part of a course, to an open source projects mimics this real-world condition in a way that is difficult, if possible at all, to re-create in a classroom setting. The one idea I had advocated was a sufficiently complex software system that was passed off from software development class to software development class, where each quarter, features were added and bugs were fixed, but… this implementation of that idea is better. Much better.
It’s heartening to see a college program that really understands this and is trying to directly address it.
I told Dave he has a good paper covering the academic aspects of using open source as a model for teaching waiting for the Journal of Higher Education.
***
After returning to the hotel, I was reading through some of the class documentation, and found out
their first assignment today—a “simple” one of “Build Firefox”—was not only due, but worth 10% of their grade.
I asked in IRC how they were being graded on the assignment. The answer is a writeup and screenshot of their locally-built browser.
I started going through a few of
the writeups, and found it very interesting. The lacking specificity of the assignment allowed prompted each student to think about it differently.
Some students grabbed the trunk from CVS and built that. Others used the source tarballs from a 1.5.0.7 release. All three major platforms were represented.
I think there’s a lot of feedback in those write-ups, detailing all the ways that we could make life easier for new contributors. And what great design for an assignment.
I had a lot of fun today.
Seneca students: I hope you do continue to wander into
#build throughout and after the course! Thanks for having me today.
mozilla, planetmoz
blahblahblah | Comments Off
V57