Posts Tagged ‘release engineering’
Tuesday, April 24th, 2012
Last week, Robert X. Cringely did a fascinating four-part series1,2,3,4 on IBM’s failing fortunes.
It’s a great series of posts, and definitely worth your time, especially if you find you and your friends pondering the Old Guard’s tactical responses to current trends in our industry at cocktail parties5.
Cringely references a quotation from an old Steve Jobs interview to explain why the problem exists, and it really resonated:
The reason IBM can’t deliver is also explained well by Steve Jobs. It’s IBM’s maniacal fixation on process, once a strength but now a cancer.
“Companies get confused,” Jobs told me. “When they start getting bigger they want to replicate their initial success. And a lot of them think well somehow there is some magic in the process of how that success was created so they start to try to institutionalize process across the company. And before very long people get very confused that the process is the content6. And that’s ultimately the downfall of IBM. IBM has the best process people in the world. They just forgot about the content.”
This is a very astute observation7 and still an incredibly common problem, even today, with respect to process. You see it in organizations of all sizes, from start-ups to, well, IBM.
I think the issue stems from the notion that if it is assumed any good process can be scaled to any size organization or problem, then the structure of the process itself must be able to be extrapolated and specified. And because of this requirement that we do the heaving lifting to extrapolate, and all the work to specify it, it will remain consistent, and thus remain effective.
It’s a tantalizing notion. If true, it would mean one can make their finely tuned, tweaked, “perfect”-process into an asset, which they can apply everywhere and/or sell.
While there might be some truth in that—certainly, many attempt to make a living doing exactly that—ultimately the structure-of-the-structure of the process8 is more important than the structure of the process.
Put another way: good organizational (build/release) process has aspects that are clearly consistent across organizational size and disciplines, but trying to bolt the steps of a enterprise corporate security firm’s release process onto, say, a desktop client for a cloud-based media product, is a recipe for a lot of consternation9.
Pre-landing checklists obviously make sense for A380s just as much as they do for 182s; good checklists in both cases will have a natural flow which is backed up by the accompanying equipment’s ergonomics, seamlessly incorporate multiple checks for critical items, and are designed to fail-as-safely-as-possible.
But they’re very different checklists.
The desire to force “wrong-sized” process is what makes everyone roll their eyes and make that “I’m gonna barf”-motion capital-P Process comes up.
Figuring out what needs to go on their own release checklists, making it logically flow between their infrastructure and processes, and fine-tuning is one of my favorite exercises with my clients10.
One size certainly does not fit all, and transplanting process wholesale seldom works. But when we dissect effective process (and process management) from other disciplines, by taking a look at its structure, not the process itself, we can learn something applicable to release engineering.
And very valuable, too.
_______________
1 Part 1
2 Part 2
3 Part 3
4 Part 4
5 As I often do…
6 Cringely later clarifies: “In this instance content means the deliverable, whether a product or service.”
7 As most of Jobs’ are…
8 How very meta!
9 This may be obvious, but I see it all the time. Usually in reverse
10 If it’s something you struggle with, we should chat!
commentary, organizational interaction, process, release engineering, strategy
Preed on Build/Release Engineering | No Comments »
V1088
Wednesday, April 11th, 2012
I finally had the chance to read the ars technica piece on Facebook’s release engineering team that’s been sitting in a tab since last week.
Despite the article’s tone being a little… Charlie-in-Wonka’s-chocolate-factory-ish, it’s got a lot of interesting tidbits, and is worth the read. It’s especially interesting to see how Facebook’s unique engineering culture has shaped their release engineering process.
(Randomly: I actually worked years ago with Chuck Rossi for a few weeks at VMware before he headed over to Google.)
I did find this bit amusing:
I eventually reached the area where the release engineering team is headquartered. Like the rest of the development personnel, release engineering uses an open space at shared tables. But their space has a unique characteristic: a well-stocked bar.
(Emphasis mine.)
Maybe Google was right all along…
agile, commentary, organizational interactions, people, release engineering, strategy
Preed on Build/Release Engineering, blahblahblah | No Comments »
V1058
Tuesday, April 3rd, 2012
My first experience with release engineering was almost fifteen years ago: I did a stint with Netscape’s release engineering team for a summer. I know I didn’t quite get why at the time, but I was hooked immediately.
My professional focus has been on build/release engineering ever since.
At various times, it’s been a difficult road to walk. I truly believe release engineering and configuration management to be an incredibly important part of the software development process, that when done right, can provide an incredible amount of value, from the smallest startups to the largest multinational corporations.
But “RelEng” is often the last part of the process to be designed, and the last part of the team to be filled. It’s often under-resourced, and at that point, it’s usually interrupt driven, (by software that needs shipping). It’s also difficult to change this trajectory, because releng is one of those things that is generally unseen and unheard, until they (or the infrastructure they’re responsible for) become the blocking factor to a ship-date.
| |
How can we approach complex and obstacle-ridden
processes safely and successfully every time? |
I think there’s a better way to approach these problems.
Friends and colleagues often giggle at me when I make repeated references to aviation in the context of release engineering. Being a pilot, I can’t help but call out the similarities as I see them. But I see so many similarities because I really do feel there is an aspect to that nature of work—whether it be air traffic control, 9-1-1 dispatch, or management of the utility grid—that we, as software engineering practitioners, can learn from and apply to make shipping software more simple, transparent, and predictable.
To that end, today, I’m launching a company: Release Engineering Approaches.
We provide a full range of build/release engineering consulting services including release engineering consulting, build engineering automation & tooling, and DevOps infrastructure support.
Our approach is to integrate those “operational” lessons and practices that so many other industries have (often painfully) learned, to make shipping whatever type of software you need to ship… as simple as possible. Every time.
If consistently and simply shipping your bits are something you struggle with, give us a shout out and let’s discuss how we can help!
To new approaches!
aviation, continuous integration, organizational interactions, processes, release engineering, strategy
Preed on Build/Release Engineering, Releases, Releng Machinery, personal | 4 Comments »
V1047
Wednesday, February 29th, 2012
Launching a 350+-ton peice of metal-loaded-with-people-and-cargo is a bit of a technical feat, not entirely unlike shipping any reasonably complex piece of software.
| |
Boeing 737 airspeed indicator
section of the
primary flight display |
That’s why those doing it day in and day out focus so much on procedures designed to increase the odds of repeatably successful outcomes.
One of these procedures is “getting the numbers,” values that directly inform pilot behavior. Numbers that don’t change are memorized. But for those that do, due to weight, locale, or weather conditions, pilots derive these values for every single takeoff1
One of these values is the so-called V1 speed; put simply, it is the speed of the aircraft during the takeoff roll at which, immaterial of circumstance, the pilot will continue the takeoff.
This is most often because this is the speed at which the length of remaining runway isn’t sufficient to safely stop the aircraft, but there could be other factors. But the important point is once this speed is called out2, even if an engine falls off, the pilot will take off.
There is an interesting analogy to release engineering: every product a software shop releases also has a “V1 speed”: the “velocity” the release reaches at which point the bits will go out the door.
This “V1 speed” usually isn’t defined by release engineers. It’s more commonly the purview of project managers, VPs, or business stakeholders3.
As a release engineer, it’s important to have an intrinsic sense of the V1 speed for each product you’re responsible for shipping. It’s one of the first values I try to calculate in new environments. Just like a pilot, it provides information that helps to inform engineering responses.
Once a sense of it is derived, working to codify the criterion that define the V1 “speed” for your release is extremely useful. I advocate such details be codified in the release checklist, in such a way that it everyone can clearly see at which point, and for what reasons, the team will abort a takeoff release, and more importantly, the reasons for which it will not4.
Exceptions, even in practice, should be rare, since encoding these criterion into your release checklist should account for serious issues-at-takeoff-release that the product can—and cannot—ship with5.
“Set takeoff thrust.” |
|
Finding your own product’s V1 speed will make every release simpler and more routine.
More importantly, it eliminates an entire class of release-related communication and coordination when things are going well, and significantly reduces the complexiy of those conversations when things aren’t.
_______________
1 If you listen to the tower frequency of any busy airport, you will invariably hear a plane request extra time to “Get the numbers.” They’re referring to this process.
2 Calling them out verbally is also part of this process; it’s hard to hear, but you can hear V1 called at 1:36
3 Optimally, of course, this “value” is an amalgum representing requirements from all groups involved in the software development process
4 Opting to ship an update instead
5 To reverse analogize, the Continental 1404 incident or the American 191 crash provide a good estimation of equivalencies for the types of catastrophic release-failures one would file under “exceptional events”; all others can reasonably be accounted for, and should be.
organizational interaction, processes, release engineering, strategy
Preed on Build/Release Engineering, Releng Machinery, tipstricks | 2 Comments »
V977
Thursday, December 15th, 2011
A friend sent me this yesterday:
Passengers on a plane are waiting for the flight to leave.
The aircraft entrance opens and two men walk up the aisle, dressed in pilot uniforms. Both are wearing dark glasses. One is using a seeing-eye dog, and the other is tapping his way up the aisle with a cane.
Nervous laughter spreads through the cabin, but the men enter the cockpit, the door closes, and the engines start.
The passengers begin glancing nervously, searching for some sign that this is just a little practical joke. None is forthcoming. The plane taxies out to the runway and begins its takeoff roll.
It moves faster and faster down the runway, and passengers near the windows realize they’re headed straight for the water at the edge of the airport.
As it begins to look as though the plane will never take off and instead plow into the water at full speed, screams of panic fill the cabin.
But at that moment, the plane lifts smoothly into the air.
Up in the cockpit, the co-pilot turns to the pilot and says, “You know, Bob, one of these days, they’re going to scream too late, and we’re all gonna die.”
This story resonated with me.
It’s probably because I’ve often made the case that successful, sustainable release engineering practices share many characteristics with (the operational nature) of aviation.
But there’s something else… something reminiscent of the many war stories we all swap about getting software shipped.
Certainly, the anecdote is meant to be absurd, and the humor lies within that absurdity. But the story also illustrates how patently obvious some problems can become when you examine them from a different perspective1.
We see how we can all be in situations where it’s easier to share nervous glances, instead of speaking up and averting panic.
And the story makes very plain how much energy we spend on angst and consternation, as we all barrel down the runway, unsure of the outcome.
While discussing these issues with a release engineer colleague, I coined the term “chicken-ship” to describe this pattern: a release involving this chicken-esque game between at least a couple2 of the release’s stakeholders, combined with the perceived threat of harm3.
As the available runway passes beneath the release, fewer and fewer have confidence it will make it off the ground; and only when that doubt turns into hysterical screaming, do we take note and do what needs to be done to avert disaster.
No one would fly if every takeoff was conducted like this, but many routinely ship release after release utilizing this “methodology.”
There’s a better way. And were we the passengers in the story, we’d already intuitively know that.
In release engineering, it may be more difficult and more subtle to see.
But it’s no less true.
_______________
1 Outside the plane!
2 Though often more
3 Which, to be fair, is often founded
aviation, organizational interaction, people, process, release engineering
Preed on Build/Release Engineering | No Comments »
V927
« Older Entries
