It occurred to me this weekend that I haven’t written about going aloft in quite some time¹, which is especially egregious given that I’m a birder these days.

Multi-hundred dollar bread-bowl of clam chowder. And Mexican Coke.

We happened to get Good Friday off², so I decided to reserve a plane for ten hours to go… somewhere, on the off chance that the weather would be good.
The weather turned out to be beautiful!
I didn’t have the time to really do any flight planning, so I ended up asking a close friend to head down to San Luis Obispo, the town of my alma mater, for a $600 bowl of clam chowder which, since I’ve flown the route more-fingers-than-I-have times, requires all of five minutes of flight planning³.
We ended up going down the coastal route instead of the inner-Salinas Valley route⁴, which provided great beach views for almost ninety minutes. Plus, I was able to enjoy the fifty-plus miles of visibility on the flight back by having my friend fly us home⁵.
My friend saw fit to capture the flight with his new camera, documenting both his expert pre-flight suggestion and a secret routine all pilots conduct before every flight, but that is an entirely unmentionable ritual of the industry.
The entire Flicker⁶ set here.
______________________________
¹ a couple of posts excluded from the count
² which is no one else in the known Government Holiday-world ™ seems to get off, but on the other hand, it’s smack dab in the middle of MLK-day and Memorial Day, so it turns out to be a better distribution of holiday-ness.
³ I mean, what’s easier than KNUQ VPLEX SNS SARDO—you want to be sure to avoid restricted-2504—PRB direct?!!
⁴ KNUQ VPLEX KMRY BSR MQO direct, roughly…
⁵ I was somewhat mean; I turned the GPS off, and said “if we get to the Oregon or Nevada borders, I’ll let ya know we’ve gone too far.”
⁶ Non-web two-point-oh spelling

One of the important roles of a build team is to help define, create, support, and use an official build environment.
In the case of a company working with a product that is open sourced, this can turn out to be a… colorful, shall we say, task, since it can often involve community members trying to build on “exotic”¹ platforms like AIX and BeOS.
MoCo uses virtualization in the build farm (and on the IT side, too) for operating systems work as guests², the result of a decision that made a lot of sense back when I was trying to get various important build systems-that-had-no-defined-environment off of shaky, failing hardware.³
As machines have gotten faster (and these days, as they have more cores), virtualization provides an easy, reasonably performant way to create, isolate, and distribute official build environments.⁴
Therefore, it wasn’t a surprise to find VMs in production roles for build-related tasks here at the Nest.
One of the common complaints about virtualization is that a completely unmodified operating system image run under reasonable (or high) workloads tend to lose time.
Virtualization may seem somewhat magical, but there’s a pretty simple reason for why this happens.⁵ In the good ol’ days, VMs tended to only lose time, but now with the 2.6 Linux kernel trying to be smart⁶, VMs can gain time, too.
In our case, a whole lotta time.
The originally reported symptom of the issue was build deliverables with timestamps far in the future. After reading up on the current fixes to the time-syncing problem (which VMware tends to rev as operating systems change), I tried a number of the standard fixes, all of which had worked for me in the past. None of them worked.
We were still gaining two minutes for every ten minutes of wall clock time. That works out to almost one-and-a-quarter days per week the VM leapt into the future.
Something was obviously, seriously wrong with this machine. I’ve been working with VMs for quite a few years now, and I’d never seen clock skew this bad.
This was a 64-bit VM, and it turns out that the kernel options you pass to fix time skew on 64-bit Linux VMs differs from the options for 32-bit. But even the “correct” options didn’t fix it.
After much Googling and forum trolling, someone suggested turning off the CPU frequency scaling daemon in the guest OS. This, of course, makes perfect sense, but the guest VM didn’t have it on.
Then it hit me: we’re using hosted virtualization. Originally, it didn’t even dawn on me to poke around the host OS’s settings, since MoCo uses the enterprise version of VMware⁷, and I was used to tackling VM problems from that angle. ESX runs VMware’s own proprietary (vm)kernel⁸, which I’m sure turns all the whizbang features on modern CPUs that would confuse the hypervisor off.
Five minutes of poking around, and I found the cause of the problem: powernowd. It was running on the host OS and “helpfully” scaling the CPU for us.
To understand why this totally screwed us, an analogy may be in order: imagine, if you will, that you must drive a car at thirty mph down a street or it’ll blow up. “Easy,” you think to yourself. But imagine that someone else is erratically controlling the gas pedal and you have to manage your speed by applying opposite braking power. This is the original virtualization timing problem.
Now imagine that the car—an eight cylinder gas guzzler—randomly decides how many cylinders to fire at any given time. It might decide to use just two, or it might decide to start using all eight.
Now what do you think your chances of maintaining a steady thirty mph would be?
A quick apt-get remove, and the next day, the builds were not… “from the future,” and the clock was still synced to the host (courtesy of vmware-guestd).
Sure, we’re using a few more electrons… but at least their deaths are for a good cause: keeping space-time from ripping apart.
In the virtual machine’s universe, at least.
____________________
¹ These days, anyway…
² [Insert complaint to Steve Jobs about lacking MacOSX Virtualization here.]
³ P2V to the rescue!
⁴ It also turns out to be cheaper and faster to have a SAN-backed, VMotion-capable build infrastructure than triply-redundant Mac Minis doing the same thing. I have to imagine it’s certainly more reliable, and less of a physical-management headache.
⁵One of the core techniques involved in virtualization is playing with clock interrupts; this means you don’t get a constant stream of clock interrupts, but rather (especially under load), you may get one or two interrupts followed by none for an interval, followed by 3 or 4 to catch you up. Of course, if you lose these interrupts over time, you’ll lose time.
⁶ The 2.6 kernels contain a few algorithms to attempt to “recover” lost clock interrupts on physical hardware. This doesn’t work in a VM, but it can cause the kernel to count interrupt that never existed, thus time moves quicker.
⁷ MoCo originally used hosted virtualization—GSX, to be specific, the product on which VMware Server is sorta-kinda based—as well; I whined (sorry Justin) until we tried out the server product.
⁸ Which, popular to contrary belief, is not a set of patches against RHEL; it merely uses RHEL to boot, and I’ve heard they can boot the vmkernel on its own in the labs in Palo Alto.

Today was my first day at Songbird¹ as the new Manager of the Build/Release Team.

San Francisco International Airport’s Runway two-eight-right Category III ILS Approach Procedure

Contrary to (popular?) irc.m.o legend, I didn’t die or get run over by a compiler.
I was, however, spending some quality (and deserved, I think) time on my couch with various DVD box sets and, post-Christmas, my new PS3.²
I must admit, even though I spent most of today setting up my workstation and email and wikis and all that (important) stuff, I’m worried I may have looked like I was a little bit out of it: I was so excited about today, I only got about ninety minutes of actual, useful, real sleep before I had to get up at 6 am.
(Oh, and now I’m finding out all the various ins and outs of CalTrain schedules. I think I need some more practice, as I totally got on the wrong train going home today. But hey, I say we all need a tour of the Silicon Valley corridor from time to time to keep our appreciation fresh.)
I’m really looking forward to the new opportunities and challenges working in an environment like Songbird presents, but I think I’m most stoked about:

• Working on a project and product related to Mozilla and Mozilla’s myriad technology.
• Once again being able to work on an open source project. I’ve always believed it’s a privilege to be able to make a living working on open source. Plus, I think as software engineers and hackers, working in that context keeps us honest about our code… or at the very least humble.
• Working on a project whose focus is on open media on the web, which I admit I’m just beginning to really feel like I grok. As an admitted die-hard Linux user who now-has-a-commute and is getting a shiny-new-iPod-for-said-commute, having an open application to get various media from various sources onto that iPod, on an open operating system, is… really important to me.

Admittedly, this is just from my sense from a single day, one with way-not-enough-sleep.
Anyway, I’ll be hanging out on IRC—even moreso now³—so if you need to find me… just holler, and you’ll get me.⁴
Still.
______________________
¹ Or Pioneers of the Inevitable—a name my mother absolutely loved when she heard it—if you prefer…
² Xbox-ers and Wii-ers: no making fun. I actually like it! And now that “Bluray has won,” I’m even more pleased with it.
³ Which admittedly wasn’t a ton the last couple months ’cause… well… come on… West Wing. And Arrested Development!
⁴ Or… Reed… one of us…