The Sober Build Engineer

A New Approach

Tuesday, April 3rd, 2012

My first experience with release engineering was almost fifteen years ago: I did a stint with Netscape’s release engineering team for a summer. I know I didn’t quite get why at the time, but I was hooked immediately.

My professional focus has been on build/release engineering ever since.

At various times, it’s been a difficult road to walk. I truly believe release engineering and configuration management to be an incredibly important part of the software development process, that when done right, can provide an incredible amount of value, from the smallest startups to the largest multinational corporations.

But “RelEng” is often the last part of the process to be designed, and the last part of the team to be filled. It’s often under-resourced, and at that point, it’s usually interrupt driven, (by software that needs shipping). It’s also difficult to change this trajectory, because releng is one of those things that is generally unseen and unheard, until they (or the infrastructure they’re responsible for) become the blocking factor to a ship-date.

How can we approach complex and obstacle-ridden processes safely and successfully every time?

I think there’s a better way to approach these problems.

Friends and colleagues often giggle at me when I make repeated references to aviation in the context of release engineering. Being a pilot, I can’t help but call out the similarities as I see them. But I see so many similarities because I really do feel there is an aspect to that nature of work—whether it be air traffic control, 9-1-1 dispatch, or management of the utility grid—that we, as software engineering practitioners, can learn from and apply to make shipping software more simple, transparent, and predictable.

To that end, today, I’m launching a company: Release Engineering Approaches.

We provide a full range of build/release engineering consulting services including release engineering consulting, build engineering automation & tooling, and DevOps infrastructure support.

Our approach is to integrate those “operational” lessons and practices that so many other industries have (often painfully) learned, to make shipping whatever type of software you need to ship… as simple as possible. Every time.

If consistently and simply shipping your bits are something you struggle with, give us a shout out and let’s discuss how we can help!

To new approaches!

Tags: aviation, continuous integration, organizational interactions, processes, release engineering, strategy
Posted in Preed on Build/Release Engineering, Releases, Releng Machinery, personal | 4 Comments »

Running the Pre-Release-Roll Numbers

Wednesday, February 29th, 2012

Launching a 350+-ton peice of metal-loaded-with-people-and-cargo is a bit of a technical feat, not entirely unlike shipping any reasonably complex piece of software.

Boeing 737 airspeed indicator section of the primary flight display

That’s why those doing it day in and day out focus so much on procedures designed to increase the odds of repeatably successful outcomes.

One of these procedures is “getting the numbers,” values that directly inform pilot behavior. Numbers that don’t change are memorized. But for those that do, due to weight, locale, or weather conditions, pilots derive these values for every single takeoff¹

One of these values is the so-called V₁ speed; put simply, it is the speed of the aircraft during the takeoff roll at which, immaterial of circumstance, the pilot will continue the takeoff.

This is most often because this is the speed at which the length of remaining runway isn’t sufficient to safely stop the aircraft, but there could be other factors. But the important point is once this speed is called out², even if an engine falls off, the pilot will take off.

There is an interesting analogy to release engineering: every product a software shop releases also has a “V₁ speed”: the “velocity” the release reaches at which point the bits will go out the door.

This “V₁ speed” usually isn’t defined by release engineers. It’s more commonly the purview of project managers, VPs, or business stakeholders³.

As a release engineer, it’s important to have an intrinsic sense of the V₁ speed for each product you’re responsible for shipping. It’s one of the first values I try to calculate in new environments. Just like a pilot, it provides information that helps to inform engineering responses.

Once a sense of it is derived, working to codify the criterion that define the V₁ “speed” for your release is extremely useful. I advocate such details be codified in the release checklist, in such a way that it everyone can clearly see at which point, and for what reasons, the team will abort a takeoff release, and more importantly, the reasons for which it will not⁴.

Exceptions, even in practice, should be rare, since encoding these criterion into your release checklist should account for serious issues-at-takeoff-release that the product can—and cannot—ship with⁵.

“Set takeoff thrust.”

Finding your own product’s V₁ speed will make every release simpler and more routine.

More importantly, it eliminates an entire class of release-related communication and coordination when things are going well, and significantly reduces the complexiy of those conversations when things aren’t.

_______________
¹ If you listen to the tower frequency of any busy airport, you will invariably hear a plane request extra time to “Get the numbers.” They’re referring to this process.
² Calling them out verbally is also part of this process; it’s hard to hear, but you can hear V₁ called at 1:36
³ Optimally, of course, this “value” is an amalgum representing requirements from all groups involved in the software development process
⁴ Opting to ship an update instead
⁵ To reverse analogize, the Continental 1404 incident or the American 191 crash provide a good estimation of equivalencies for the types of catastrophic release-failures one would file under “exceptional events”; all others can reasonably be accounted for, and should be.

Tags: organizational interaction, processes, release engineering, strategy
Posted in Preed on Build/Release Engineering, Releng Machinery, tipstricks | 2 Comments »

Agile (With a Capital-A)

Tuesday, February 21st, 2012

Late last year, I had the opportunity to attend Agile/Scrum training led by one of the experts in the field, Kenny Rubin.

Like many developers and software shops, I’ve done “agile [lowercase 'a'] software development” in the past, but had never had a rigorous explanation of the details of Agile, so the day-long training offering a proper treatment of the topic was both long overdue for me and welcomed.

Rubin’s training starts with a walk through of the common problems we’ve all faced as developers. As he catalogs these problems, the group begins to make the case to themselves that waterfall and other classic methods no longer serve the creation of software product very well¹. This helps to frame the entire day, as Rubin returns to the conclusion the group brought itself to whenever discussions arise about whether Agile will really work²

It turns out a day is only enough to cover the highlights of Agile, so in addition to the treatment of why, Rubin’s training focused on the who—the makeup of the scrum team—and the what—the sprint process—with multiple minor forays into the various hows, with a healthy dose of gotchas.

Having had a few weeks now to absorb the material, as well as look back on various ways I’ve practiced it myself, the day of training raised some interesting issues for me:

1. As a release engineer, how do you effectively integrate build/release engineering and processes into the Agile framework? This may seem like a stupid question, but I have yet to see it entirely effectively integrated into an agile-shop, a fact Rubin was nice enough to discuss with a colleague and myself over lunch.
   
   Rubin suggested that release engineers be made part of the scrum team, so they’re integrated into the Agile process. This is an obvious solution, but it presents an issue for larger teams: if you have 8-10 scrum teams, then assigning a release engineer to each team’s daily scrum processes is likely to mean your release engineer will spend large parts of his days running between scrum meetings. It also doesn’t leave much resourcing for handling the interrupt-driven nature of release engineering. Rubin also describes one aspect of the scrum team being its self-organizing nature; most developers won’t normally involve release engineers³
   
   Having said that, I don’t mean to imply that Rubin is incorrect in his suggestion, merely that given the levels at which I’ve seen most release engineering teams staffed, it would be difficult to support that model.
   
   “Hire more release engineers” may be implicit in his suggestion, a notion I will seldom argue with.
2. In all the ways I’ve seen agile practiced, there is one aspect that Rubin discussed at length, which I’ve never really see play out the way he says it’s supposed to: as a balance to engineering committing to finishing a set of work in a particular sprint, the product owner commits to not introducing change in a sprint, and does not question or “game” the costing of tasks or the filling of the sprint work-item list. It would be great it worked like this (and I’m sure that Agile works much better when all sides agree to this), but I can conjure up example after example where requirements where changed inside the sprint, and engineers were either pushed to revise the estimate of a backlog item⁵ or add more to the sprint than they felt comfortable⁶.
   
   Without these checks and balances, it’s obviously difficult to realize the full benefits of capital-a Agile, but 50+ years of software development has positioned product owners to not expect any checks, balances, or limitations placed upon their requests⁷
3. With Agile, it’s easy to fall into the trap of bastardizing Agile to a point where it’s no longer a viable software development methodology, to say nothing of “Agile”. An aspect of the extreme programming/agile “fad” that became very popular years ago was something I called “cafeteria software development process”: take whatever tenets of Agile work for you, and leave the rest.
   
   The problem with this trap is we tend to take the aspects that work for certain parts of the organization—costing, breaking work down into sprints, etc.—but ignore the parts that are harder to sell and implement (no change after sprint starts, for instance).
   
   Probably the most important thing I learned from Rubin is that Agile actually means something. It’s a set of values about software development that are embodied in a set of activities and processes that seeks to make software development more successful. It has aspects that are flexible (length of sprints, for instance), but other aspects aren’t fungible (the check and balance between engineering and product owner). When you remove those aspects, what you’re doing isn’t Agile anymore and, in fact, may not even be agile. All too many software shops have a product back log, do sprints, and maybe even use an Agile tracking tool. But not all parts of the organization have committed to Agile, so they really aren’t an Agile shop.
   
   

Having seen a lot of software shops that “do agile” (and being a part of a few myself), Rubin’s training was not only clear and eminently understandable, but refreshing and has provided a lot of food for thought.

He cut through a lot of the hype about what Agile is, what it is not, and what organizations, from individuals to teams, need to do to structure their interactions and work to create an Agile environment, and succeed using it.

My only complaint about the training was that it was only a day long.

_______________
¹ If they ever did
² As a trainer, I’m willing to bet it also has the great side effect of giving Rubin insight into what notions line-engineers have about Agile, and what topics he may need to address more directly. It’s a great training technique.
³ My professional experience has been this failure to do so is generally due to omission rather than, say, maliciousness^4
4 Though, I have experienced the latter from time to time, unfortunately
⁵ “You don’t reallythink it’s going to take that long, do you?,” with the associated performance review implication such a question implies
⁶ “You can do more than that in two weeks. I know it!”
⁷ A fact a software engineering Robert Glass also explores

Tags: agile, processes
Posted in Preed on Build/Release Engineering | 4 Comments »

Caught in the Space/Build Continuum, Part II

Monday, October 3rd, 2011

Last week, I wrote about a tool I conjured up to help describe the various positions organizations have their build teams in: Preed’s Build Team Spectrum.

I wanted to talk a bit more about the X-axis. (I’ll leave the Y-axis for another post.)

The biggest source of problems (and it was precisely this that caused Brad¹ and I to talk past each other in our discussion) is when expectations—whether they be between a build engineer and her manager, the release team manager and other managers, or the build team and the engineering organization as a whole—differ.

You might imagine this wouldn’t happen, but it happens all the time.

As I explained to Brad: “When someone hired build or QA engineers, the majority of the time², the description and mandate of the role is not ‘make developers more productive.’ It’s ’ship/QA’ the product.”

Of course, making developers successful, as Brad initially pointed out, really should be at the core of any engineering support organization. But often, there is release engineering technical debt that the organization must pay off to get to a sustainable place before the focus tends back to developer support. Some of the worst, ugliest, most demoralizing situations I’ve seen in my career have been due to mismatches in these exact expectations.

It’s gummy hard being caught in the middle…

For instance, another release engineer used to tell me how he kept getting pestered about why the new release tool hadn’t been finished, but his manager never put together that they were doing the releases manually, and had faced 2-3 months of continued emergency-patch releases.

His manager thought he was on the right side of the X-axis; he was actually spending his days in the center, shipping bits manually. (He was in the red zone to boot, and quit soon thereafter.)

It’s a common story: I’ve experienced situations where developers and their managers become angry when I can’t immediately respond to their support requests: they want me on the left side of the X-axis, because no one ever told them that the build team had been chronically understaffed and the mandate we were given was to keep shipping—that’s a constant—and keep the build farm from keeling over. Developer support was always going to be “the next build engineer we hire”-’s responsibility.

Again, mismatched perceptions.

Making it clear which resources are available on which portions of the spectrum is an incredibly important facet of keeping the build team productive and engaged with the larger engineering organization.

When everyone is honest with each other and on the same page about which resources are available for what types of activities, and those current mandates, positions, and expectations are broadly communicated and time taken for them to be understood, you have will happier engineers and managers, no matter what team they’re on.

_______________
¹ My college friend who’s still actually probably not named that
² Though admittedly, not always

Tags: organizational interaction, people, processes
Posted in Preed on Build/Release Engineering, Releng Machinery | No Comments »

Caught in the Space/Build Continuum, Part I

Thursday, September 29th, 2011

Editor’s note: original post was split into two parts; part II will get published on Monday.

I was catching up with an old college-buddy of mine, Brad¹, recently and we started swapping work stories. Since he’s a software engineer, we (predictably) stumbled onto the topic of build engineering.

He was frustrated at work: the build tools weren’t what he would’ve chosen, builds took too long to build (and rebuild), adding new modules was a pain, and his build team seemed content to ignore the problems. He struggled to understand why the build engineers weren’t more helpful: “We spend the entire project lifetime working on the build tree, whereas the product ships just once, at the end,” he reasoned.

“Wouldn’t it make more sense,” he continued, “to dedicate more resources to making our lives easier? It seems like it would surely pay back in increased developer efficiency.”

I certainly know where he was coming from: one place I worked, you used to be able to get a good game of foosball in between incremental rebuilds². And while it always killed me, I explained to Brad why it was never fixed: “If I went to a VP of Engineering and said ‘Well, we can’t ship the product today, because I spent my time making [some subset of] your developers more productive,’ he’d fire me. And he’d be totally right to do so; developers are my secondary customers.”

He retorted: “If we actually accounted for all of the time we spent rebuilding broken trees, recovering from dependency-fails, waiting for branches to get fixed, etc. etc. etc. you’d get fired too. And he’d be totally right to do so; without the developers, after all, there is no product.”

While I was slightly miffed by the sentiment, it’s nothing I hadn’t heard before. And, in the strictest interpretation, he was, of course, right.

But I thought about it some more and I realized we were both right. And we were both wrong.

Every environment I’ve worked in has entailed a mix of “developer support” and “build infrastructure construction/maintenance.” I took a hard line position to make a point³, but Brad’s expectations of the build team weren’t right either: it’s always been a continuum^4,5.

If I were to plot this continuum, it’d look something like this:

Apologies in advance to the absolutely wonderful Indexed.
Click to big-ify.

Some areas of note:

• Note that the Shipping Software band takes up the middle of the X-axis, but as more is invested in build infrastructure, this band will actually shrink.
• The Build Guru’s⁶ sole responsibility is to monitor continuous integration, work with developers to fix failures, and generally provide interrupt-driven services for developer/program managers. Usually seen in bigger teams, with big budgets.
• Crank Builders and Turners work on the “crank” that releases your software; some environments find it useful to make this distinction; the separation’s success largely depends on management and personalities involved, I’ve found.
• The area of the spectrum marked by poor decisions is where the build team has such little influence and the environment is so developer-driven that the act of shipping the bits is negatively impacted⁷.
• Someone who turns out to be a surprisingly good Source of Great Specs is engaged in developer support, but has the ability to influence those processes, which often gives the role on the other side of the graph a lot of help in the requirements for a build infrastructure that can directly and positively impact developer productivity.
• The “Possible Warning Zone” exists in any environment where the build team has maximal influence, but is disconnected from the developer-support function. It’s not a given: a build team focusing on requirements solicitation from the engineering organization can help provide an almost-utopian environment for developers; if they team doesn’t, it’s painful. The good news: I have never seen a build team in this position.
• The “Bad Process Feeding Zone” marks conditions where, given the correct circumstances, poor process flourishes, either by creation of new, bad process, or the stagnation of old, bad process.
• Build Engineer Burnout Central marks the zone where the organization will just burn through people left and right: mindless turning of the ineffective crank, with a focus on nothing put pushing software out the door. Good engineers will get bored and leave. Poor engineers will get stressed out. And leave.

So we have a nice graph to describe where build engineers focus their time and effort, and now we can discuss where they are, where they might move to support the organizational needs and what issues that may result as that all happens.

I’ll talk more about this X-axis, and why Brad and I were both simultaneously right and wrong on Monday.

_______________
¹ That’s probably not his real name
² Two if you were lucky or bad at the game
³ I had prefaced our conversation with this caveat
⁴ Which he admitted afterward as well
⁵ It’s one of things I actually enjoy about release engineering
⁶ This was a term we used at VMware; I’ve heard the role dubbed other things
⁷ A recent example: the complications that ensued for those switching to distributed version control systems without any real plan; someone said “This is great, we’re doing it,” and it was done

Tags: organizational interactions, people, processes
Posted in Preed on Build/Release Engineering, Releng Machinery | 2 Comments »