Continuous Integration

Release Schedule

May 2020
S M T W T F S
« Nov    
 12
3456789
10111213141516
17181920212223
24252627282930
31  

Branches

Twitter

© 2020 The Sober Build Engineer | Simply ship. Every time.

Simply ship. Every time.

“This is going on your permanent record”

01/24/2012

Major media outlets have started picking up Google’s news today that they’d be aggregating user profile and usage data across all of their products.

To be honest, I didn’t find this particularly surprising since I assumed Google had long been doing this.

But the winning quotation from the press release was this nugget:

Our recently launched personal search feature is a good example of the cool things Google can do when we combine information across products.

“Cool?”

Cool?!

Google “Search Plus Your World ™” was characterized in many ways when it shipped.

Of all The Web’s sentiment I saw, “cool” was not among it.

Unless you by “cool,” you mean “incredibly creepy, of limited actual use, and full of chilling effects.”

Does this change fall under “doing evil?”

I suppose it depends on how they end up using the data (and whether or not any independent entity could ever audit those statements to assert truth)… but however they use it, I’m pretty sure it will result in some pretty epic unintended consequences.

The Sobering Posts of 2011

12/30/2011

One of my favorite tech writers, @rands, recently wrote up a year in review, and it inspired me to spend some time thinking about 2011 myself.

One goal for the year was to write more consistently about release engineering, my experiences, and its evolving role in software development. I certainly did write more, and I’m pleased with that, but it wasn’t as consistent as I’d hoped. A resolution for 2012, I suppose.

In any event, these are the favorite (and popular) Sober Build Engineer posts of 2011:

Getting Back on Message: My commitment to myself to writing more really began with finding my voice again. (Plus, this post includes a delightfully horrible photo of a teenage-me!)

The Elevator Storyteller explains why I constantly return to aviation when analyzing release and process engineering.

Not Everything Is The Circus challenged the seemingly prevailing notion that everything we do in software development must be “fun” and—more disconcertingly— if it’s not “fun,” we can simply ignore it. Very flawed logic.

Introducing QuickRelease: With Jobs’ passing this year, we’ve heard his [in]famous “real artists ship”-quip over and over again; I shipped QuickRelease this year. I’m very proud of that, and I’m looking forward to executing more on QuickRelease’s roadmap in 2012.

Caught in the Space/Build Continuum, Parts I and II was the result of a great (and ongoing!) conversation with a college friend-now-developer, and tried to close the gap between what developers, program managers, and QA think release engineers do, and what release engineers really spend their day doing… (Part I is probably my favorite post of the year; the graph really illustrates the explanation, I think.)

I had the honor of chatting with Perforce about, among other things, release engineering, DVCS, P4 Streams, and the role of a build engineer. A significantly cuter-looking photo of me was included in that interview…

The Chicken-Ship Method wonders why the process of shipping software by playing veiled, glorified games of chicken with each other is still so common.

2012 is going to be an interesting year, I think, both for the industry and for the evolution of release engineering. I think the last few years have seen the introduction of a lot of interesting, new technologies, and we’ve seen those used in both successful and less-than-successful ways. And as we push more deployment and release abilities to the developer, we’ll grapple with where the appropriate line is, and how that impacts roles, responsibilities, and accountability.

And I’ll still be here, writing about it all.

Have a happy and (safe) New Years!

The Chicken-Ship Method

12/15/2011

A friend sent me this yesterday:

Passengers on a plane are waiting for the flight to leave.

The aircraft entrance opens and two men walk up the aisle, dressed in pilot uniforms. Both are wearing dark glasses. One is using a seeing-eye dog, and the other is tapping his way up the aisle with a cane.

Nervous laughter spreads through the cabin, but the men enter the cockpit, the door closes, and the engines start.

The passengers begin glancing nervously, searching for some sign that this is just a little practical joke. None is forthcoming. The plane taxies out to the runway and begins its takeoff roll.

It moves faster and faster down the runway, and passengers near the windows realize they’re headed straight for the water at the edge of the airport.

As it begins to look as though the plane will never take off and instead plow into the water at full speed, screams of panic fill the cabin.

But at that moment, the plane lifts smoothly into the air.

Up in the cockpit, the co-pilot turns to the pilot and says, “You know, Bob, one of these days, they’re going to scream too late, and we’re all gonna die.”

This story resonated with me.

It’s probably because I’ve often made the case that successful, sustainable release engineering practices share many characteristics with (the operational nature) of aviation.

But there’s something else… something reminiscent of the many war stories we all swap about getting software shipped.

Certainly, the anecdote is meant to be absurd, and the humor lies within that absurdity. But the story also illustrates how patently obvious some problems can become when you examine them from a different perspective1.

We see how we can all be in situations where it’s easier to share nervous glances, instead of speaking up and averting panic.

And the story makes very plain how much energy we spend on angst and consternation, as we all barrel down the runway, unsure of the outcome.

While discussing these issues with a release engineer colleague, I coined the term “chicken-ship” to describe this pattern: a release involving this chicken-esque game between at least a couple2 of the release’s stakeholders, combined with the perceived threat of harm3.

As the available runway passes beneath the release, fewer and fewer have confidence it will make it off the ground; and only when that doubt turns into hysterical screaming, do we take note and do what needs to be done to avert disaster.

No one would fly if every takeoff was conducted like this, but many routinely ship release after release utilizing this “methodology.”

There’s a better way. And were we the passengers in the story, we’d already intuitively know that.

In release engineering, it may be more difficult and more subtle to see.

But it’s no less true.

_______________
1 Outside the plane!
2 Though often more
3 Which, to be fair, is often founded

Managing Mail Madness with Mutt

12/12/2011

A few weeks ago, I tweeted about a great blog post on a technique for managing the massive amounts of email many of us navigate daily.

I’d become frustrated with the state of my own inbox1 and had been on the prowl for a new method to manage the email torrent.

Having used it for a little over a week now, I can say: it works wonders!

One caveat: the author uses Postbox, which has some core features that are the cornerstone of the technique.

Now, I’ve been a longtime proponent of Postbox, and I use it as my work client. But for a variety of historical reasons, my personal email-life is centered around the venerable Mutt mail client2. So, the technique described in the article works perfectly if you use Postbox.

This is how I successfully re-created the work flow with Mutt.

Labeling

The technique relies heavily on the ability to label mail, as Gmail and Postbox both easily and handily support.

Mutt calls this “tagging,” and while there’s been support for reading the X-Label header since 1.4, support to edit labels isn’t in a currently-released Mutt.

The good news: there’s a patch to support label/tag editing.

The bad news: the patch is against a pretty old version of Mutt3.

I’m a Gentoo user, so I updated the patch4 to apply to [Gentoo's] 1.5.205,6 and also wrote an ebuild8 you can add into a local overlay to make it easier to install9.

After that, add the following to your muttrc to bind label-editing to keys and display labels:

# Add X-Label to your unignored headers
unignore From: To: Subject: Date: Cc: Reply-To: X-Label:
color header brightwhite default '^X-Label:'
macro index Cy " ~y " "Limit view to label"

The patch binds the ‘y’ key to adding, deleting, or changing a message label in both Mutt’s message and index views; the above config binds Ctrl-y to limiting the view of the folder based on a specific label.

When editing labels, they are space-separated.

Search, Don’t Sort

The “Search, don’t sort”-mantra isn’t new; but when all you have is grep, you still tend to sort your email as an initial step to being able to easily find it later.

Enter mairix10.

Setup is relatively easy: configure it with an rc file, and run it. I run it from cron every four hours to keep the index fresh, and reset the entire index every month11

Since mairix provides its results as an mbox or Maildir folder, I wrote a simple wrapper that only launches Mutt if there are results to my search and makes the folder read-only, to keep the conceptual model—these are search results, and I shouldn’t edit them—consistent.

As for sorting, I still do sort some email, mostly Twitter, Facebook, and other automated notifications, as well as mail from important people that tends to fit neatly into boundaries (my parents, certain friends, etc.)

Keeping Folders Tidy

Now that my mail is searchable, where we save it isn’t as important. Now I can move to a model of just saving everything to mutt’s default saved-messages and going from there.

Only problem is, won’t that folder balloon over time? Probably.

Cron comes to our rescue again; I have the following entry:

5 0 1 * * MAILBOX="/home/preed/mail/saved-messages-archive/saved-messages-$(/bin/date +%Y-%m)" && /usr/bin/find /home/preed/mail/ -maxdepth 1 -type l -name 'saved-messages' -exec rm {} ; && /bin/touch $MAILBOX && /bin/ln -s $MAILBOX /home/preed/mail/saved-messages

This makes “saved-messages” a symlink that automatically gets updated at 12:05 am every month to point to a folder with the year and month.

And mairix updates its index with these new folders the index gets reset every month.

I use this same technique for my sent-messages, spamassassin-populated spam folder, and deleted-messages folder; the last two are auto-deleted by cron six months later.

Filtering Assaults On Your Inbox

The last tenent of this email management technique is probably the most important: if you don’t want it in your inbox again, set up a filter immediately. Don’t wait.

I’d been getting spam mails from Ticketmaster forevar. I didn’t set up a filter for them until I started using this technique.
It’s a bit costly at first, but it pays off and quicker than you think.

To help with this, I set up an easy way to test procmail rules without running them on my live inbox.

I stole the scripts from this page and slightly modified them to fit paths on my system.

Inbox Zero?!!

After using this method for about a week, I was able to dump most of my inbox into a saved-messages-archive folder, and begin using the dated saved-messages folder.

I’ve been able to find any message I’ve needed to find, and my inbox now hovers around 50 messages.

Finally, people emailing me are getting responses on the order of 72 hours; before, if their message got lost in my inbox, it was hit and miss; I just restarted a four month old thread, because I’d lost track of it, and couldn’t tag it “todo.”

Have I reached Inbox Zero? No. But as the original post points out: “I don’t care about ‘Inbox Zero’ because there’ll be more [email] tomorrow as soon as the sun comes up.”

But using this technique, I’ve been able to turn my inbox into something manageable, without the fear12 that I’ll lose a message.

All in all, I’ve been very impressed with how it’s been working so far. Hats off to Mr. Ignition for the inspiration!

_______________
1 Over 2,000 messages in 75 megabytes before I started this project
2 Having used them since I was 13, my customized Mutt key bindings, cribbed from Pine, are hopelessly burned into my muscle memory
3 It’s against 1.5.1; the current released version is 1.5.21
4 PGP sig
5 Which I’m using because of an attachment handling bug in 1.5.21
6 I also hacked up patches against pristine-1.5.20 (PGP sig) and pristine-1.5.217 (PGP sig)
7 This patch doesn’t include the documentation changes; that’s left as an exercise for the reader
8 I just added one to the ebuild revision; I’d love to hear from a Gentoo-dev on whether this is the correct way to do this, because I’m betting it’s not…
9 There’s also a method to allow editing of labels using an external script, but I didn’t use this script, so I can’t vouch for how well it works
10 Which I have @scanlime to thank for turning me on to
11 A full index of my 1.6 gigabytes of mail takes 97 seconds; reindexes only look at new messages, so it’s pretty low cost
12 Which was probably largely irrational anyway

QuickRelease 0.12: shipped!

11/07/2011

Just a quick (pun intended) heads up for QuickRelease followers/users: 0.12 just shipped today!

You can grab an automatically-generated tarball, courtesy of github, here.

The 0.12 release has a bunch of minor fixes and polishes to the 0.11.1 release. Notable improvements include:

  • Finish up a wholesale refactor of “Partner steps”: partner steps were originally created to model release processes that are relatively close to each other, but contain minor configuration differences for different partners. Specific improvements include:
    • Creation of a new top-level type: PartnerStep
    • Move all the partner-related concepts into this class.
    • Use a cleaner approach to connect the running of Processes and (Partner)Steps
    • Hook up auto-setting of the partner-related configuration steps, so partner-specific configuration is all set up for steps that use them
    • Provide a mechanism to get at partner-specific configuration variables in common config areas
    • Allow partner steps to decide whether or not they should halt when encountering an error or keep trudging along.
  • Hook up the console driver’s “ignore any errors” option (-i) actually work, and also correctly report if an error occurred, even if you’re ignoring it.
  • Create a collection type for ReleaseFrameworkErrors&mdash:ReleaseFrameworkErrorCollection—which allows errors to be “batched up” and dealt with (particular useful when writing partner steps.
  • Plus a bunch of minor bug fixes!

The provided examples have also been updated to illustrate the power of PartnerSteps and of the new QuickRelease ConfigSpec magic you can play with.

Please give play arond with it, and if you have any questions or (heaven forbid) discover any bugs, feel free to let the QuickRelease developers Google group know!

“[Hack on] it and They will come”

11/01/2011

From a great Apple story:

Once again, my sanity was saved by the kindness of a stranger. At 2:00 one morning, a visitor appeared in my office: the engineer responsible for making the PowerPC system disk master. He explained things this way: “Apple is a hardware company. There are factories far away building Apple computers. One of the final steps of their assembly line is to copy all of the system software from the ‘Golden Master’ hard disk onto each computer’s hard disk. I create the Golden Master and FedEx it to the manufacturing plant. In a very real and pragmatic sense, I decide what software does and does not ship.” He told me that if I gave him our software the day before the production run began, it could appear on the Golden Master disk.

I’d bet money the mysterious engineer that saved the day was a release engineer.

This is a minor historical footnote in Apple’s history, but it’s a fascinating look at the policies, environment, camaraderie, and engineering spirit of the company in its previous life; definitely worth a few minutes.

(Via @othiym23, originally via @raum.)

***

I must admit: one of the things I do miss the most is creating the golden masters. I’m young enough to have not had to worry about it for years, but old enough to have had to do it for a few products that I shipped.

I think we might have a different mentality about software release process and quality if we still went through that ritual today, and it actually involved cardboard boxes, pieces of plastic to hold the bits, and shipping trucks.

Caught in the Space/Build Continuum, Part II

10/03/2011

Last week, I wrote about a tool I conjured up to help describe the various positions organizations have their build teams in: Preed’s Build Team Spectrum.

I wanted to talk a bit more about the X-axis. (I’ll leave the Y-axis for another post.)

The biggest source of problems (and it was precisely this that caused Brad1 and I to talk past each other in our discussion) is when expectations—whether they be between a build engineer and her manager, the release team manager and other managers, or the build team and the engineering organization as a whole—differ.

You might imagine this wouldn’t happen, but it happens all the time.

As I explained to Brad: “When someone hired build or QA engineers, the majority of the time2, the description and mandate of the role is not ‘make developers more productive.’ It’s ‘ship/QA’ the product.”

Of course, making developers successful, as Brad initially pointed out, really should be at the core of any engineering support organization. But often, there is release engineering technical debt that the organization must pay off to get to a sustainable place before the focus tends back to developer support. Some of the worst, ugliest, most demoralizing situations I’ve seen in my career have been due to mismatches in these exact expectations.


It’s gummy hard being caught in the middle…

For instance, another release engineer used to tell me how he kept getting pestered about why the new release tool hadn’t been finished, but his manager never put together that they were doing the releases manually, and had faced 2-3 months of continued emergency-patch releases.

His manager thought he was on the right side of the X-axis; he was actually spending his days in the center, shipping bits manually. (He was in the red zone to boot, and quit soon thereafter.)

It’s a common story: I’ve experienced situations where developers and their managers become angry when I can’t immediately respond to their support requests: they want me on the left side of the X-axis, because no one ever told them that the build team had been chronically understaffed and the mandate we were given was to keep shipping—that’s a constant—and keep the build farm from keeling over. Developer support was always going to be “the next build engineer we hire”-’s responsibility.

Again, mismatched perceptions.

Making it clear which resources are available on which portions of the spectrum is an incredibly important facet of keeping the build team productive and engaged with the larger engineering organization.

When everyone is honest with each other and on the same page about which resources are available for what types of activities, and those current mandates, positions, and expectations are broadly communicated and time taken for them to be understood, you have will happier engineers and managers, no matter what team they’re on.

_______________
1 My college friend who’s still actually probably not named that
2 Though admittedly, not always

Caught in the Space/Build Continuum, Part I

09/29/2011

Editor’s note: original post was split into two parts; part II will get published on Monday.

I was catching up with an old college-buddy of mine, Brad1, recently and we started swapping work stories. Since he’s a software engineer, we (predictably) stumbled onto the topic of build engineering.

He was frustrated at work: the build tools weren’t what he would’ve chosen, builds took too long to build (and rebuild), adding new modules was a pain, and his build team seemed content to ignore the problems. He struggled to understand why the build engineers weren’t more helpful: “We spend the entire project lifetime working on the build tree, whereas the product ships just once, at the end,” he reasoned.

“Wouldn’t it make more sense,” he continued, “to dedicate more resources to making our lives easier? It seems like it would surely pay back in increased developer efficiency.”

I certainly know where he was coming from: one place I worked, you used to be able to get a good game of foosball in between incremental rebuilds2. And while it always killed me, I explained to Brad why it was never fixed: “If I went to a VP of Engineering and said ‘Well, we can’t ship the product today, because I spent my time making [some subset of] your developers more productive,’ he’d fire me. And he’d be totally right to do so; developers are my secondary customers.”

He retorted: “If we actually accounted for all of the time we spent rebuilding broken trees, recovering from dependency-fails, waiting for branches to get fixed, etc. etc. etc. you’d get fired too. And he’d be totally right to do so; without the developers, after all, there is no product.”

While I was slightly miffed by the sentiment, it’s nothing I hadn’t heard before. And, in the strictest interpretation, he was, of course, right.

But I thought about it some more and I realized we were both right. And we were both wrong.

Every environment I’ve worked in has entailed a mix of “developer support” and “build infrastructure construction/maintenance.” I took a hard line position to make a point3, but Brad’s expectations of the build team weren’t right either: it’s always been a continuum4,5.

If I were to plot this continuum, it’d look something like this:


Apologies in advance to the absolutely wonderful Indexed.
Click to big-ify.

Some areas of note:

  • Note that the Shipping Software band takes up the middle of the X-axis, but as more is invested in build infrastructure, this band will actually shrink.
  • The Build Guru‘s6 sole responsibility is to monitor continuous integration, work with developers to fix failures, and generally provide interrupt-driven services for developer/program managers. Usually seen in bigger teams, with big budgets.
  • Crank Builders and Turners work on the “crank” that releases your software; some environments find it useful to make this distinction; the separation’s success largely depends on management and personalities involved, I’ve found.
  • The area of the spectrum marked by poor decisions is where the build team has such little influence and the environment is so developer-driven that the act of shipping the bits is negatively impacted7.
  • Someone who turns out to be a surprisingly good Source of Great Specs is engaged in developer support, but has the ability to influence those processes, which often gives the role on the other side of the graph a lot of help in the requirements for a build infrastructure that can directly and positively impact developer productivity.
  • The “Possible Warning Zone” exists in any environment where the build team has maximal influence, but is disconnected from the developer-support function. It’s not a given: a build team focusing on requirements solicitation from the engineering organization can help provide an almost-utopian environment for developers; if they team doesn’t, it’s painful. The good news: I have never seen a build team in this position.
  • The “Bad Process Feeding Zone” marks conditions where, given the correct circumstances, poor process flourishes, either by creation of new, bad process, or the stagnation of old, bad process.
  • Build Engineer Burnout Central marks the zone where the organization will just burn through people left and right: mindless turning of the ineffective crank, with a focus on nothing put pushing software out the door. Good engineers will get bored and leave. Poor engineers will get stressed out. And leave.

So we have a nice graph to describe where build engineers focus their time and effort, and now we can discuss where they are, where they might move to support the organizational needs and what issues that may result as that all happens.

I’ll talk more about this X-axis, and why Brad and I were both simultaneously right and wrong on Monday.

_______________
1 That’s probably not his real name
2 Two if you were lucky or bad at the game
3 I had prefaced our conversation with this caveat
4 Which he admitted afterward as well
5 It’s one of things I actually enjoy about release engineering
6 This was a term we used at VMware; I’ve heard the role dubbed other things
7 A recent example: the complications that ensued for those switching to distributed version control systems without any real plan; someone said “This is great, we’re doing it,” and it was done

A Farm Without A Farmer

09/13/2011

Songbird’s build farm recently moved nests.

This I know because an ex-coworker recently asked for some random configuration details and any gotchas to look out for, in preparation for the migration1.

After we finished discussing “bidness,” I wondered aloud how the farm had fared, post-Preed as the shepherd. His answer surprised me: it had been humming along fine for months. In fact, he said they hadn’t really needed to touch a thing until now.

I was, of course, glad to hear that the infrastructure I built years ago had continued to “Just Work” ™ for them, so their main activity—shipping code—wasn’t affected in the slightest.

But it was also nice to have real-world data to validate my approach to building and managing the farm.

The core methods of the approach include:

  • Being meticulous about the core of your organization’s build infrastructure—the build farm—matters. As do the details. Thinking about them as a bunch of machines that “just have stuff installed” on them is a recipe for disaster.
  • Taking time to design the infrastructure is important. Specifically, we made the infrastructure pluggable, so the VM images were common, but certain features were tunable. This allowed different VMs to be used for different things, but the images weren’t entirely dissimilar or unrelated to each other2.
  • Access control to the farm is important. Shared passwords and multiple access paths are your enemy. You need to know and manage who can access which resources, and what they have access to inside the farm, so you can…
  • Implement some process for basic change control. This is a core tenet of configuration management, but for some reason, it is often the first thing to fall by the wayside. Sometimes, this will require being a bit of a Nazi about it, but it pays off for the entire organization, even developers.
  • You need to build visibility into the farm operations. Whether it’s Nagios, ESX monitoring, or even just the way continuous integration notifications are handled, building in early-reporting of problems is key to keeping a farm humming

While these all may seem simple, it of course turns out to be more complex to implement. And I’m sure that the farm’s months of happy compiling was helped by a lower rate of configuration changes.

But this provides proof that if you make the time, you can sow the seeds of a build farm that is reliable, stable, allows you to “Set it and forget it3,” and will provide months (at least!) of builds crops.

_______________
1 I was more than happy to help ensure a successful flight
2 For example, the artifact for each VM were stored on a separate drive, so machines responsible for larger builds could easily have larger drives and wouldn’t run out of space, and VMs doing smaller builds weren’t wasting space
3 Thank you Ronco!

Documentation Review: Words That Work

09/12/2011

I first saw Dr. Frank Luntz on a 2008 episode of Bill Maher’s Real Time1.

Ever since the interview and the subsequent panel discussion, I’d been referring to the concepts in his book—Words That Work—despite having never having actually read it. I decided it was past time I corrected that2.

Luntz’s message in Words That Work is extremely simple: it’s not what you say, it’s what people hear.

That’s it.

It’s so central to the book, he repeats it twice in his introduction and closes the book with it. You’ll read it a total of 26 times, almost twice per chapter.

In addition to mere repetition, he explores this point through statistics he’s gathered over years of research on America’s populace, through a lens as both consumer and voter.

While it may seem like a simple idea, Luntz regails us with tale after tale of companies, executives and candidates who forgot or ignored this simple idea.

His other major point, the one I more often referred to: “Single words really do control the debate.” Luntz explains:

In effect, positioning an idea doesn’t merely “frame” it so that it carries a certain meaning; it actually defines the terms of the debate itself.

Perhaps unsurprisingly, there were more than a few gems which I thought readily applied to release engineering3:

The language lesson: Focus on results, not process.

If it’s really true that good release engineering is one part politics and two part communication, with a pinch of coding thrown in here and there, understanding these insights is certainly useful.

One bit of Luntz’s approach that may stir up some cynicism among readers is his (thinly veiled) political leanings. He spends the good part of a chapter describing how his techniques apply to all ends of the political spectrum, and that it’s important for anyone engaging in a debate to set aside their position for a moment and try to understand the fundamental points about communication he’s trying to make.

He’s totally right.

But when you look at his examples—penning the Contract With America, writing the pro ballot argument for Gray Davis’ recall, working with accounting firms in the wake of the Enron scandals, contorting the “estate tax” into the “death tax”—it’s obvious who his clients are and who they’re targeting. The numerous and constant examples left a little voice wondering whether his conclusions were biased by the conservative slant in which much of his research has been conducted.

Having said that, it’s hard to contend that his major points—”it’s not what you say, it’s what people hear” and framing the debate matters—are incorrect.

And while some of his case studies may make you cringe (and may require some serious translation to how they would apply to a situation that doesn’t involve voting for a Republican candidate), Luntz’s book provides useful tips to improve your ability to make sure the message your audience is hearing is what you’re intending, whether it’s in a convention hall or a conference room.

_______________
1 He was a fidgety interview, and seemed more interested in telling Maher not to touch him than in answering the questions
2 I’ve committed this particular sin with The Checklist Manifesto, which I’m also currently remedying
3 Or any role that involves basic political skills

Newer Posts
Older Posts