February 3rd, 2012
@cheeseplus1 pointed me to a post this week by the always-engaging2 Bruce Schneier covering yet-another Transportation Security Agency snafu.
Schneier’s summation of the event:
- TSA screener finds two pipes3 in passenger’s bags.
- Screener determines that they’re not a threat.
- Screener confiscates them anyway, because of their “material and appearance.”
- Because they’re not actually a threat, screener leaves them at the checkpoint.
- Everyone forgets about them.
- Six hours later, the next shift of TSA screeners notices the pipes and — not being able to explain how they got there and, presumably, because of their “material and appearance” — calls the police bomb squad to remove the pipes.
- TSA does not evacuate the airport, or even close the checkpoint, because — well, we don’t know why.
I noted how this felt like there was a strong analogy to be made to how some release engineering teams go about their work.
But in the moment, there was just a nagging sense of that. I spent some more time thinking about why, exactly, this was so reminiscent:
- Conceptual Inconsistency: The screener initially deemed the items were not a threat. But wait, then they were. So they were confiscated. But then they weren’t actually treated like a loaded gun4 or remediated in any meaningful way.
These types of cognitive inconsistencies present a real problem for others (especially engineers) trying to assess the reasoning behind procedures and process. In the TSA’s case, “because we said so” is a reason that sticks. It’s a less valid reason in the majority of other venues, including the one we operate in.
When we’re inconsistent, we provide so much ammo to be taken less seriously. Or ignored completely.
- Queue management: presumably the reason the screener left the (supposedly) dangerous items lingering at the checkpoint was because he became distracted. Like a lot of release engineering tasks, screening passengers is inherently interrupt-driven. The fact that the security checkpoint’s operations are structured such that a (common and constant) interruption was allowed to, in effect, compromise the entire purpose of the checkpoint, indicates that there’s a queue management issue that needs addressing.
- Follow through: So obviously the ba, er pipes were dropped when they weren’t dealt with as any other threat. What is especially interesting, though, is that when the initial inconsistency was detected, the TSA didn’t follow its own procedure for addressing the problem. This, again, creates an inconsistency and credibility issue, in addition to the obviously embarrassing operational one.
At the end of the day, these types of (seeingly ongoing) occurrences only serve to reduce the credibility of the TSA, and support the argument that it’s all just security theater that offers no real value or benefit.
The effects of those same credibility-reducing actions represent a very real issue for build teams operating in similar ways: when events like this happens, it’s much easier for others to assume that all we’re engaged in is “release engineering theater.”
It’s an important lesson about a class of traps we need to be conscious and vigilant of avoiding, and need to design procedures that inherently address them.
And then commit to consistently executing those procedures.
_______________
1 Who can argue with a sentiment involving more cheese?
2 And often amusing
3 Pipes? Really? This feels like a GTA mission already
4 Y’know, any other threat
Join the Conversation: No Comments »
V953
January 24th, 2012
Major media outlets have started picking up Google’s news today that they’d be aggregating user profile and usage data across all of their products.
To be honest, I didn’t find this particularly surprising since I assumed Google had long been doing this.
But the winning quotation from the press release was this nugget:
Our recently launched personal search feature is a good example of the cool things Google can do when we combine information across products.
“Cool?”
Cool?!
Google “Search Plus Your World ™” was characterized in many ways when it shipped.
Of all The Web’s sentiment I saw, “cool” was not among it.
Unless you by “cool,” you mean “incredibly creepy, of limited actual use, and full of chilling effects.”
Does this change fall under “doing evil?”
I suppose it depends on how they end up using the data (and whether or not any independent entity could ever audit those statements to assert truth)… but however they use it, I’m pretty sure it will result in some pretty epic unintended consequences.
Join the Conversation: 1 Comment »
V944
December 30th, 2011
One of my favorite tech writers, @rands, recently wrote up a year in review, and it inspired me to spend some time thinking about 2011 myself.
One goal for the year was to write more consistently about release engineering, my experiences, and its evolving role in software development. I certainly did write more, and I’m pleased with that, but it wasn’t as consistent as I’d hoped. A resolution for 2012, I suppose.
In any event, these are the favorite (and popular) Sober Build Engineer posts of 2011:
Getting Back on Message: My commitment to myself to writing more really began with finding my voice again. (Plus, this post includes a delightfully horrible photo of a teenage-me!)
The Elevator Storyteller explains why I constantly return to aviation when analyzing release and process engineering.
Not Everything Is The Circus challenged the seemingly prevailing notion that everything we do in software development must be “fun” and—more disconcertingly— if it’s not “fun,” we can simply ignore it. Very flawed logic.
Introducing QuickRelease: With Jobs’ passing this year, we’ve heard his [in]famous “real artists ship”-quip over and over again; I shipped QuickRelease this year. I’m very proud of that, and I’m looking forward to executing more on QuickRelease’s roadmap in 2012.
Caught in the Space/Build Continuum, Parts I and II was the result of a great (and ongoing!) conversation with a college friend-now-developer, and tried to close the gap between what developers, program managers, and QA think release engineers do, and what release engineers really spend their day doing… (Part I is probably my favorite post of the year; the graph really illustrates the explanation, I think.)
I had the honor of chatting with Perforce about, among other things, release engineering, DVCS, P4 Streams, and the role of a build engineer. A significantly cuter-looking photo of me was included in that interview…
The Chicken-Ship Method wonders why the process of shipping software by playing veiled, glorified games of chicken with each other is still so common.
2012 is going to be an interesting year, I think, both for the industry and for the evolution of release engineering. I think the last few years have seen the introduction of a lot of interesting, new technologies, and we’ve seen those used in both successful and less-than-successful ways. And as we push more deployment and release abilities to the developer, we’ll grapple with where the appropriate line is, and how that impacts roles, responsibilities, and accountability.
And I’ll still be here, writing about it all.
Have a happy and (safe) New Years!
Join the Conversation: 1 Comment »
V932
December 15th, 2011
A friend sent me this yesterday:
Passengers on a plane are waiting for the flight to leave.
The aircraft entrance opens and two men walk up the aisle, dressed in pilot uniforms. Both are wearing dark glasses. One is using a seeing-eye dog, and the other is tapping his way up the aisle with a cane.
Nervous laughter spreads through the cabin, but the men enter the cockpit, the door closes, and the engines start.
The passengers begin glancing nervously, searching for some sign that this is just a little practical joke. None is forthcoming. The plane taxies out to the runway and begins its takeoff roll.
It moves faster and faster down the runway, and passengers near the windows realize they’re headed straight for the water at the edge of the airport.
As it begins to look as though the plane will never take off and instead plow into the water at full speed, screams of panic fill the cabin.
But at that moment, the plane lifts smoothly into the air.
Up in the cockpit, the co-pilot turns to the pilot and says, “You know, Bob, one of these days, they’re going to scream too late, and we’re all gonna die.”
This story resonated with me.
It’s probably because I’ve often made the case that successful, sustainable release engineering practices share many characteristics with (the operational nature) of aviation.
But there’s something else… something reminiscent of the many war stories we all swap about getting software shipped.
Certainly, the anecdote is meant to be absurd, and the humor lies within that absurdity. But the story also illustrates how patently obvious some problems can become when you examine them from a different perspective1.
We see how we can all be in situations where it’s easier to share nervous glances, instead of speaking up and averting panic.
And the story makes very plain how much energy we spend on angst and consternation, as we all barrel down the runway, unsure of the outcome.
While discussing these issues with a release engineer colleague, I coined the term “chicken-ship” to describe this pattern: a release involving this chicken-esque game between at least a couple2 of the release’s stakeholders, combined with the perceived threat of harm3.
As the available runway passes beneath the release, fewer and fewer have confidence it will make it off the ground; and only when that doubt turns into hysterical screaming, do we take note and do what needs to be done to avert disaster.
No one would fly if every takeoff was conducted like this, but many routinely ship release after release utilizing this “methodology.”
There’s a better way. And were we the passengers in the story, we’d already intuitively know that.
In release engineering, it may be more difficult and more subtle to see.
But it’s no less true.
_______________
1 Outside the plane!
2 Though often more
3 Which, to be fair, is often founded
Join the Conversation: No Comments »
V927
December 12th, 2011
A few weeks ago, I tweeted about a great blog post on a technique for managing the massive amounts of email many of us navigate daily.
I’d become frustrated with the state of my own inbox1 and had been on the prowl for a new method to manage the email torrent.
Having used it for a little over a week now, I can say: it works wonders!
One caveat: the author uses Postbox, which has some core features that are the cornerstone of the technique.
Now, I’ve been a longtime proponent of Postbox, and I use it as my work client. But for a variety of historical reasons, my personal email-life is centered around the venerable Mutt mail client2. So, the technique described in the article works perfectly if you use Postbox.
This is how I successfully re-created the work flow with Mutt.
Labeling
The technique relies heavily on the ability to label mail, as Gmail and Postbox both easily and handily support.
Mutt calls this “tagging,” and while there’s been support for reading the X-Label header since 1.4, support to edit labels isn’t in a currently-released Mutt.
The good news: there’s a patch to support label/tag editing.
The bad news: the patch is against a pretty old version of Mutt3.
I’m a Gentoo user, so I updated the patch4 to apply to [Gentoo's] 1.5.205,6 and also wrote an ebuild8 you can add into a local overlay to make it easier to install9.
After that, add the following to your muttrc to bind label-editing to keys and display labels:
# Add X-Label to your unignored headers
unignore From: To: Subject: Date: Cc: Reply-To: X-Label:
color header brightwhite default '^X-Label:'
macro index \Cy "
~y " "Limit view to label"
The patch binds the ‘y’ key to adding, deleting, or changing a message label in both Mutt’s message and index views; the above config binds Ctrl-y to limiting the view of the folder based on a specific label.
When editing labels, they are space-separated.
Search, Don’t Sort
The “Search, don’t sort”-mantra isn’t new; but when all you have is grep, you still tend to sort your email as an initial step to being able to easily find it later.
Enter mairix10.
Setup is relatively easy: configure it with an rc file, and run it. I run it from cron every four hours to keep the index fresh, and reset the entire index every month11
Since mairix provides its results as an mbox or Maildir folder, I wrote a simple wrapper that only launches Mutt if there are results to my search and makes the folder read-only, to keep the conceptual model—these are search results, and I shouldn’t edit them—consistent.
As for sorting, I still do sort some email, mostly Twitter, Facebook, and other automated notifications, as well as mail from important people that tends to fit neatly into boundaries (my parents, certain friends, etc.)
Keeping Folders Tidy
Now that my mail is searchable, where we save it isn’t as important. Now I can move to a model of just saving everything to mutt’s default saved-messages and going from there.
Only problem is, won’t that folder balloon over time? Probably.
Cron comes to our rescue again; I have the following entry:
5 0 1 * * MAILBOX="/home/preed/mail/saved-messages-archive/saved-messages-$(/bin/date +\%Y-\%m)" && /usr/bin/find /home/preed/mail/ -maxdepth 1 -type l -name 'saved-messages' -exec rm {} \; && /bin/touch $MAILBOX && /bin/ln -s $MAILBOX /home/preed/mail/saved-messages
This makes “saved-messages” a symlink that automatically gets updated at 12:05 am every month to point to a folder with the year and month.
And mairix updates its index with these new folders the index gets reset every month.
I use this same technique for my sent-messages, spamassassin-populated spam folder, and deleted-messages folder; the last two are auto-deleted by cron six months later.
Filtering Assaults On Your Inbox
The last tenent of this email management technique is probably the most important: if you don’t want it in your inbox again, set up a filter immediately. Don’t wait.
I’d been getting spam mails from Ticketmaster forevar. I didn’t set up a filter for them until I started using this technique.
It’s a bit costly at first, but it pays off and quicker than you think.
To help with this, I set up an easy way to test procmail rules without running them on my live inbox.
I stole the scripts from this page and slightly modified them to fit paths on my system.
Inbox Zero?!!
After using this method for about a week, I was able to dump most of my inbox into a saved-messages-archive folder, and begin using the dated saved-messages folder.
I’ve been able to find any message I’ve needed to find, and my inbox now hovers around 50 messages.
Finally, people emailing me are getting responses on the order of 72 hours; before, if their message got lost in my inbox, it was hit and miss; I just restarted a four month old thread, because I’d lost track of it, and couldn’t tag it “todo.”
Have I reached Inbox Zero? No. But as the original post points out: “I don’t care about ‘Inbox Zero’ because there’ll be more [email] tomorrow as soon as the sun comes up.”
But using this technique, I’ve been able to turn my inbox into something manageable, without the fear12 that I’ll lose a message.
All in all, I’ve been very impressed with how it’s been working so far. Hats off to Mr. Ignition for the inspiration!
_______________
1 Over 2,000 messages in 75 megabytes before I started this project
2 Having used them since I was 13, my customized Mutt key bindings, cribbed from Pine, are hopelessly burned into my muscle memory
3 It’s against 1.5.1; the current released version is 1.5.21
4 PGP sig
5 Which I’m using because of an attachment handling bug in 1.5.21
6 I also hacked up patches against pristine-1.5.20 (PGP sig) and pristine-1.5.217 (PGP sig)
7 This patch doesn’t include the documentation changes; that’s left as an exercise for the reader
8 I just added one to the ebuild revision; I’d love to hear from a Gentoo-dev on whether this is the correct way to do this, because I’m betting it’s not…
9 There’s also a method to allow editing of labels using an external script, but I didn’t use this script, so I can’t vouch for how well it works
10 Which I have @scanlime to thank for turning me on to
11 A full index of my 1.6 gigabytes of mail takes 97 seconds; reindexes only look at new messages, so it’s pretty low cost
12 Which was probably largely irrational anyway
Join the Conversation: No Comments »
V919
