The Widest Common Denominator

07/28/2010

One of the common issues raised on my build system¹ code reviews is why I went to the effort to add hooks in a particular place or make certain things variables instead of just hard-coding the values.

“No one will ever change that,” the reviewing developer argues.

The ensuing explanation always reminds me of a story a mentor of mine once told me:

The scene: a meeting between various high-level engineering managers, including a few of the company’s founders², and of course my mentor, the build/release team lead.

The topic? Whether or not rewrite the aging build system… and if so, how?

With an increasing list of products, we’d outgrown the build system currently driving the builds³

In the course of the meeting, various developer pain-points with the current system were vented; efficiency problems⁴ were detailed; limitations on the release-side, which were starting to affect business decisions were described.

Eventually, everyone reached the same conclusion: this was one rewrite that really was necessary. Excitement from those who fought daily with the current system permeated the room, like a nerdy kid in a candy store being told “Hey, grab whatever you want and make an über-treat!”

In the way only an engineer noticing their opportunity to really solve a problem they understand⁵ can, discussion shifted immediately to implementation details: what should the new system be responsible for? What should the interfaces be? What are the high level components? What language should we use? [Should we reinvent the wheel?] Who should write it? How will we deploy it?

Granted, it was totally out of scope for the meeting, but as you might expect, such discussion was of much more interest to the engineers.

One of the issues raised was how one should be able to invoke the shiny, new, to-be-implemented build system. One of the engineers in the room said “Oh, that’s easy; you’ll just run this command; done.”

My mentor raised his hand and said “Well, if we’re going to hook this into the release tools, we’ll need to be able to call it in these couple of ways, since the current system supports that and we make heavy use of it.”

The engineer replied “No, we don’t need to support that; you can just change the interface in the tools; it’s really easy.”

Another engineer blurted out “No, we need that; I do my daily builds by using those targets. Besides, the performance team hooks in by calling that other target, too.”

The two engineers started arguing with each other about whether or not their use cases were valid. The conversation drew the other engineers in the room into the fray.

Like a panel from a fight in one of those old Peanut comics trips, statements like “You build the product by doing that? Really?!! Why on god’s green earth would you ever do it that way” were heard to be uttered, as some of the smartest engineers in the company argued with each other about how they did their own builds,

My mentor, quiet as a mouse, just sat back in his chair… and smiled.

I can only imagine the smirk on his face was due largely to the fact that the discussion had turned into an argument about individual work-flow preferences. Those are values differences, which try though one might, an engineer won’t come to a “correct” solution.

Because there isn’t one.

This is why so often in build system-related code, one will see myriad ./configure options despite the default being used 98% of the time; why the call to that common utility program can be overridden by the environment; why there are “prologue” and “fin” hooks that can be plugged into.

Good build engineers internalize early on in their career⁶ the necessity for this sort of code.

It’s not “over-engineering” or “useless complexity.” That target that makes no sense to you? We added that for the developer who’s now the CTO. The environmental override? Yah, that allowed one my colleague to get our product building on PH-UX in a a couple of days instead of three weeks. Those weird “pre-” and “post-” hooks? The QA team uses them for automation. And the performance testing team. Oh, and us. We use those to do, y’know, releases.

These “structured layers of indirection” address the requirement that a build/release organization support the maximum number of developer and QA use-cases it reasonably can, as often as it is able to do so; they are such a ubiquitous solution, they’re probably better characterized as a “build system design pattern⁷.”

To this day, when I find myself reminded of my mentor’s anecdote while explaining this pattern manifested in code, I often long for another engineer to randomly walk by, innocently eavesdropping at just the right point to blurt out to the other developer “Wait, you want to do what with our build system?!”

It really is the most tangible, likely-to-stick explanation available.

And it’s certainly more amusing for me.

_______________
¹ Think makefiles and scripts used during the build itself
² All still hard-core developers within the organization
³ Given that system had been in place since the company was the founders’ “summer project”, no one could claim it didn’t have a good run…
⁴ When a complete build of the product takes 5 or 6 hours, and is trending in the wrong direction, everyone starts paying attention to this
⁵ Or so they think…
⁶ Assuming they make a career of it
⁷ More on… well… more of these later. Hopefully…