Posts Tagged ‘people’
Wednesday, April 11th, 2012
I finally had the chance to read the ars technica piece on Facebook’s release engineering team that’s been sitting in a tab since last week.
Despite the article’s tone being a little… Charlie-in-Wonka’s-chocolate-factory-ish, it’s got a lot of interesting tidbits, and is worth the read. It’s especially interesting to see how Facebook’s unique engineering culture has shaped their release engineering process.
(Randomly: I actually worked years ago with Chuck Rossi for a few weeks at VMware before he headed over to Google.)
I did find this bit amusing:
I eventually reached the area where the release engineering team is headquartered. Like the rest of the development personnel, release engineering uses an open space at shared tables. But their space has a unique characteristic: a well-stocked bar.
(Emphasis mine.)
Maybe Google was right all along…
agile, commentary, organizational interactions, people, release engineering, strategy
Preed on Build/Release Engineering, blahblahblah | No Comments »
V1058
Thursday, December 15th, 2011
A friend sent me this yesterday:
Passengers on a plane are waiting for the flight to leave.
The aircraft entrance opens and two men walk up the aisle, dressed in pilot uniforms. Both are wearing dark glasses. One is using a seeing-eye dog, and the other is tapping his way up the aisle with a cane.
Nervous laughter spreads through the cabin, but the men enter the cockpit, the door closes, and the engines start.
The passengers begin glancing nervously, searching for some sign that this is just a little practical joke. None is forthcoming. The plane taxies out to the runway and begins its takeoff roll.
It moves faster and faster down the runway, and passengers near the windows realize they’re headed straight for the water at the edge of the airport.
As it begins to look as though the plane will never take off and instead plow into the water at full speed, screams of panic fill the cabin.
But at that moment, the plane lifts smoothly into the air.
Up in the cockpit, the co-pilot turns to the pilot and says, “You know, Bob, one of these days, they’re going to scream too late, and we’re all gonna die.”
This story resonated with me.
It’s probably because I’ve often made the case that successful, sustainable release engineering practices share many characteristics with (the operational nature) of aviation.
But there’s something else… something reminiscent of the many war stories we all swap about getting software shipped.
Certainly, the anecdote is meant to be absurd, and the humor lies within that absurdity. But the story also illustrates how patently obvious some problems can become when you examine them from a different perspective1.
We see how we can all be in situations where it’s easier to share nervous glances, instead of speaking up and averting panic.
And the story makes very plain how much energy we spend on angst and consternation, as we all barrel down the runway, unsure of the outcome.
While discussing these issues with a release engineer colleague, I coined the term “chicken-ship” to describe this pattern: a release involving this chicken-esque game between at least a couple2 of the release’s stakeholders, combined with the perceived threat of harm3.
As the available runway passes beneath the release, fewer and fewer have confidence it will make it off the ground; and only when that doubt turns into hysterical screaming, do we take note and do what needs to be done to avert disaster.
No one would fly if every takeoff was conducted like this, but many routinely ship release after release utilizing this “methodology.”
There’s a better way. And were we the passengers in the story, we’d already intuitively know that.
In release engineering, it may be more difficult and more subtle to see.
But it’s no less true.
_______________
1 Outside the plane!
2 Though often more
3 Which, to be fair, is often founded
aviation, organizational interaction, people, process, release engineering
Preed on Build/Release Engineering | No Comments »
V927
Monday, October 3rd, 2011
Last week, I wrote about a tool I conjured up to help describe the various positions organizations have their build teams in: Preed’s Build Team Spectrum.
I wanted to talk a bit more about the X-axis. (I’ll leave the Y-axis for another post.)
The biggest source of problems (and it was precisely this that caused Brad1 and I to talk past each other in our discussion) is when expectations—whether they be between a build engineer and her manager, the release team manager and other managers, or the build team and the engineering organization as a whole—differ.
You might imagine this wouldn’t happen, but it happens all the time.
As I explained to Brad: “When someone hired build or QA engineers, the majority of the time2, the description and mandate of the role is not ‘make developers more productive.’ It’s ’ship/QA’ the product.”
Of course, making developers successful, as Brad initially pointed out, really should be at the core of any engineering support organization. But often, there is release engineering technical debt that the organization must pay off to get to a sustainable place before the focus tends back to developer support. Some of the worst, ugliest, most demoralizing situations I’ve seen in my career have been due to mismatches in these exact expectations.
It’s gummy hard being caught in the middle…
For instance, another release engineer used to tell me how he kept getting pestered about why the new release tool hadn’t been finished, but his manager never put together that they were doing the releases manually, and had faced 2-3 months of continued emergency-patch releases.
His manager thought he was on the right side of the X-axis; he was actually spending his days in the center, shipping bits manually. (He was in the red zone to boot, and quit soon thereafter.)
It’s a common story: I’ve experienced situations where developers and their managers become angry when I can’t immediately respond to their support requests: they want me on the left side of the X-axis, because no one ever told them that the build team had been chronically understaffed and the mandate we were given was to keep shipping—that’s a constant—and keep the build farm from keeling over. Developer support was always going to be “the next build engineer we hire”-’s responsibility.
Again, mismatched perceptions.
Making it clear which resources are available on which portions of the spectrum is an incredibly important facet of keeping the build team productive and engaged with the larger engineering organization.
When everyone is honest with each other and on the same page about which resources are available for what types of activities, and those current mandates, positions, and expectations are broadly communicated and time taken for them to be understood, you have will happier engineers and managers, no matter what team they’re on.
_______________
1 My college friend who’s still actually probably not named that
2 Though admittedly, not always
organizational interaction, people, processes
Preed on Build/Release Engineering, Releng Machinery | No Comments »
V896
Thursday, September 29th, 2011
Editor’s note: original post was split into two parts; part II will get published on Monday.
I was catching up with an old college-buddy of mine, Brad1, recently and we started swapping work stories. Since he’s a software engineer, we (predictably) stumbled onto the topic of build engineering.
He was frustrated at work: the build tools weren’t what he would’ve chosen, builds took too long to build (and rebuild), adding new modules was a pain, and his build team seemed content to ignore the problems. He struggled to understand why the build engineers weren’t more helpful: “We spend the entire project lifetime working on the build tree, whereas the product ships just once, at the end,” he reasoned.
“Wouldn’t it make more sense,” he continued, “to dedicate more resources to making our lives easier? It seems like it would surely pay back in increased developer efficiency.”
I certainly know where he was coming from: one place I worked, you used to be able to get a good game of foosball in between incremental rebuilds2. And while it always killed me, I explained to Brad why it was never fixed: “If I went to a VP of Engineering and said ‘Well, we can’t ship the product today, because I spent my time making [some subset of] your developers more productive,’ he’d fire me. And he’d be totally right to do so; developers are my secondary customers.”
He retorted: “If we actually accounted for all of the time we spent rebuilding broken trees, recovering from dependency-fails, waiting for branches to get fixed, etc. etc. etc. you’d get fired too. And he’d be totally right to do so; without the developers, after all, there is no product.”
While I was slightly miffed by the sentiment, it’s nothing I hadn’t heard before. And, in the strictest interpretation, he was, of course, right.
But I thought about it some more and I realized we were both right. And we were both wrong.
Every environment I’ve worked in has entailed a mix of “developer support” and “build infrastructure construction/maintenance.” I took a hard line position to make a point3, but Brad’s expectations of the build team weren’t right either: it’s always been a continuum4,5.
If I were to plot this continuum, it’d look something like this:
Apologies in advance to the absolutely wonderful Indexed.Click to big-ify.
Some areas of note:
- Note that the Shipping Software band takes up the middle of the X-axis, but as more is invested in build infrastructure, this band will actually shrink.
- The Build Guru’s6 sole responsibility is to monitor continuous integration, work with developers to fix failures, and generally provide interrupt-driven services for developer/program managers. Usually seen in bigger teams, with big budgets.
- Crank Builders and Turners work on the “crank” that releases your software; some environments find it useful to make this distinction; the separation’s success largely depends on management and personalities involved, I’ve found.
- The area of the spectrum marked by poor decisions is where the build team has such little influence and the environment is so developer-driven that the act of shipping the bits is negatively impacted7.
- Someone who turns out to be a surprisingly good Source of Great Specs is engaged in developer support, but has the ability to influence those processes, which often gives the role on the other side of the graph a lot of help in the requirements for a build infrastructure that can directly and positively impact developer productivity.
- The “Possible Warning Zone” exists in any environment where the build team has maximal influence, but is disconnected from the developer-support function. It’s not a given: a build team focusing on requirements solicitation from the engineering organization can help provide an almost-utopian environment for developers; if they team doesn’t, it’s painful. The good news: I have never seen a build team in this position.
- The “Bad Process Feeding Zone” marks conditions where, given the correct circumstances, poor process flourishes, either by creation of new, bad process, or the stagnation of old, bad process.
- Build Engineer Burnout Central marks the zone where the organization will just burn through people left and right: mindless turning of the ineffective crank, with a focus on nothing put pushing software out the door. Good engineers will get bored and leave. Poor engineers will get stressed out. And leave.
So we have a nice graph to describe where build engineers focus their time and effort, and now we can discuss where they are, where they might move to support the organizational needs and what issues that may result as that all happens.
I’ll talk more about this X-axis, and why Brad and I were both simultaneously right and wrong on Monday.
_______________
1 That’s probably not his real name
2 Two if you were lucky or bad at the game
3 I had prefaced our conversation with this caveat
4 Which he admitted afterward as well
5 It’s one of things I actually enjoy about release engineering
6 This was a term we used at VMware; I’ve heard the role dubbed other things
7 A recent example: the complications that ensued for those switching to distributed version control systems without any real plan; someone said “This is great, we’re doing it,” and it was done
organizational interactions, people, processes
Preed on Build/Release Engineering, Releng Machinery | 2 Comments »
V878
