Build System Improvements By the Wall Clock Numbers


It’s like a Turbo button…
only faster!12

In between the constant grind of getting releases out the door, it’s often a struggle to find time to make improvements that are imminently noticeable.
That’s not to say that us build engineers are sitting in a room, typing make && cp over and over again and staring at the screen until it finishes.2

On the contrary, improvements to our build and release infrastructure are made constantly.

But they often involve making our lives easier: a script that handles errors more appropriately, or hooking up two pieces of the puzzle together, so we can go to bed at 2 am, instead of 4 am. These are not readily visible to the end-user or our customers, the developers3.
So, given that, you’ll have to forgive me for being a bit squee4 about the following:

svn up -r 13127 | grep ^Updated &&
time make -f > /dev/null 2>&1 &&
make -f clobber > /dev/null 2>&1  &&
svn up -r 13128 | grep ^Updated &&
time make -f > /dev/null 2>&1
Updated to revision 13127.
real  3m45.008s
user  3m7.301s
sys   0m26.064s
Updated to revision 13128.
real  2m40.991s
user  2m22.129s
sys   0m17.253s

Yes, that really is a 29% improvement in wall-clock build time.

On repeated trials, it drops to about 26%. I’m guessing this has to do with caching the source files.

And these results are from Linux; since the core problem7 had to do with excessive use of $(shell ...), and therefore fork(), the results should be even slightly better on Win32.8 I didn’t test Mac, but one of our Mac gurus twittered the Turbo before I mentioned it to anyone.

Lessons learned:

  1. make‘s $(shell ...) construct is very useful. It’s also extremely dangerous, in part because it’s so convenient, easy to use, and… useful. If you can do anything else to avoid it, do that.
  2. If you’re assigning the results of a $(shell ...) call into a variable, you probably want :=, not =. And if you really want =, see LL#1.
  3. Win32 MSYS fork()ing performance has a notoriously bad reputation… but it’s not like fork(2) is free on Linux or Mac. I’m surprised at the savings on Linux and the small delta between Linux and Win32.
  4. Running make SHELL="/bin/sh +x"9 on your project might prove very interesting to you. Are you really executing that much extra fluff in a subshell? Yes. Yes you are.10
  5. Make has a surprising number of useful functions; if you’re spending any time doing build system stuff, read up on them. I myself was guilty of doing ifeq (exists,$(shell test -e $(FILE) && echo exists) before I learned about $(wildcard ...). The former spawns three processes. The latter: zero.

This is just one of the few makeovers11 that I’m working on; there’s something in the works for the installer, and I’m working on some pretty fundamental changes to the build system itself.

(I call that project Build SystemNG, or BSNG for short.)

I’d be lying if I said I didn’t enjoy working on something that people notice… either because it’s noticeably faster or completely broken.

One of the two.
1the famed “rebuild the airplane while it’s in flight”-problem, which I’ve decided is probably the most asinine analogy I’ve ever heard. So I’m going to stop using it.
2 As some might have you believe
3 Except, maybe, if you’re paying attention to the amount of scotch consumed
4 I keep thinking everyone knows what this means, but I keep running into people who don’t. Even people who I would think would… don’t. So here’s a convenia-link5 for it to Urban Dictionary, in case you don’t.
6 Like a perma-link, only less permanent, and more convenient
7 Bug 15529 has all the details
8 I only had a VM to test those; I don’t have the numbers in front of me, but as I remember, they were somewhere between 30 and 32%.
9 A benchmarking hint from Mecklenberg’s O’Reilly GMake book, which I recommend
10Over half a million of them, actually… which is why tips-and-tricks are always helpful: everyone wants faster builds.
11 Hahahahhaa… make over? gmake? Get it?
12 Photo courtesy D.G.S..