Caught in the Space/Build Continuum, Part I

09/29/2011

Editor’s note: original post was split into two parts; part II will get published on Monday.

I was catching up with an old college-buddy of mine, Brad¹, recently and we started swapping work stories. Since he’s a software engineer, we (predictably) stumbled onto the topic of build engineering.

He was frustrated at work: the build tools weren’t what he would’ve chosen, builds took too long to build (and rebuild), adding new modules was a pain, and his build team seemed content to ignore the problems. He struggled to understand why the build engineers weren’t more helpful: “We spend the entire project lifetime working on the build tree, whereas the product ships just once, at the end,” he reasoned.

“Wouldn’t it make more sense,” he continued, “to dedicate more resources to making our lives easier? It seems like it would surely pay back in increased developer efficiency.”

I certainly know where he was coming from: one place I worked, you used to be able to get a good game of foosball in between incremental rebuilds². And while it always killed me, I explained to Brad why it was never fixed: “If I went to a VP of Engineering and said ‘Well, we can’t ship the product today, because I spent my time making [some subset of] your developers more productive,’ he’d fire me. And he’d be totally right to do so; developers are my secondary customers.”

He retorted: “If we actually accounted for all of the time we spent rebuilding broken trees, recovering from dependency-fails, waiting for branches to get fixed, etc. etc. etc. you’d get fired too. And he’d be totally right to do so; without the developers, after all, there is no product.”

While I was slightly miffed by the sentiment, it’s nothing I hadn’t heard before. And, in the strictest interpretation, he was, of course, right.

But I thought about it some more and I realized we were both right. And we were both wrong.

Every environment I’ve worked in has entailed a mix of “developer support” and “build infrastructure construction/maintenance.” I took a hard line position to make a point³, but Brad’s expectations of the build team weren’t right either: it’s always been a continuum^4,5.

If I were to plot this continuum, it’d look something like this:

Apologies in advance to the absolutely wonderful Indexed.
Click to big-ify.

Some areas of note:

• Note that the Shipping Software band takes up the middle of the X-axis, but as more is invested in build infrastructure, this band will actually shrink.
• The Build Guru‘s⁶ sole responsibility is to monitor continuous integration, work with developers to fix failures, and generally provide interrupt-driven services for developer/program managers. Usually seen in bigger teams, with big budgets.
• Crank Builders and Turners work on the “crank” that releases your software; some environments find it useful to make this distinction; the separation’s success largely depends on management and personalities involved, I’ve found.
• The area of the spectrum marked by poor decisions is where the build team has such little influence and the environment is so developer-driven that the act of shipping the bits is negatively impacted⁷.
• Someone who turns out to be a surprisingly good Source of Great Specs is engaged in developer support, but has the ability to influence those processes, which often gives the role on the other side of the graph a lot of help in the requirements for a build infrastructure that can directly and positively impact developer productivity.
• The “Possible Warning Zone” exists in any environment where the build team has maximal influence, but is disconnected from the developer-support function. It’s not a given: a build team focusing on requirements solicitation from the engineering organization can help provide an almost-utopian environment for developers; if they team doesn’t, it’s painful. The good news: I have never seen a build team in this position.
• The “Bad Process Feeding Zone” marks conditions where, given the correct circumstances, poor process flourishes, either by creation of new, bad process, or the stagnation of old, bad process.
• Build Engineer Burnout Central marks the zone where the organization will just burn through people left and right: mindless turning of the ineffective crank, with a focus on nothing put pushing software out the door. Good engineers will get bored and leave. Poor engineers will get stressed out. And leave.

So we have a nice graph to describe where build engineers focus their time and effort, and now we can discuss where they are, where they might move to support the organizational needs and what issues that may result as that all happens.

I’ll talk more about this X-axis, and why Brad and I were both simultaneously right and wrong on Monday.

_______________
¹ That’s probably not his real name
² Two if you were lucky or bad at the game
³ I had prefaced our conversation with this caveat
⁴ Which he admitted afterward as well
⁵ It’s one of things I actually enjoy about release engineering
⁶ This was a term we used at VMware; I’ve heard the role dubbed other things
⁷ A recent example: the complications that ensued for those switching to distributed version control systems without any real plan; someone said “This is great, we’re doing it,” and it was done