Wednesday, April 11th, 2012
I finally had the chance to read the ars technica piece on Facebook’s release engineering team that’s been sitting in a tab since last week.
Despite the article’s tone being a little… Charlie-in-Wonka’s-chocolate-factory-ish, it’s got a lot of interesting tidbits, and is worth the read. It’s especially interesting to see how Facebook’s unique engineering culture has shaped their release engineering process.
(Randomly: I actually worked years ago with Chuck Rossi for a few weeks at VMware before he headed over to Google.)
I did find this bit amusing:
I eventually reached the area where the release engineering team is headquartered. Like the rest of the development personnel, release engineering uses an open space at shared tables. But their space has a unique characteristic: a well-stocked bar.
(Emphasis mine.)
Maybe Google was right all along…
agile, commentary, organizational interactions, people, release engineering, strategy
Preed on Build/Release Engineering, blahblahblah | No Comments »
V1058
Tuesday, April 3rd, 2012
My first experience with release engineering was almost fifteen years ago: I did a stint with Netscape’s release engineering team for a summer. I know I didn’t quite get why at the time, but I was hooked immediately.
My professional focus has been on build/release engineering ever since.
At various times, it’s been a difficult road to walk. I truly believe release engineering and configuration management to be an incredibly important part of the software development process, that when done right, can provide an incredible amount of value, from the smallest startups to the largest multinational corporations.
But “RelEng” is often the last part of the process to be designed, and the last part of the team to be filled. It’s often under-resourced, and at that point, it’s usually interrupt driven, (by software that needs shipping). It’s also difficult to change this trajectory, because releng is one of those things that is generally unseen and unheard, until they (or the infrastructure they’re responsible for) become the blocking factor to a ship-date.
| |
How can we approach complex and obstacle-ridden
processes safely and successfully every time? |
I think there’s a better way to approach these problems.
Friends and colleagues often giggle at me when I make repeated references to aviation in the context of release engineering. Being a pilot, I can’t help but call out the similarities as I see them. But I see so many similarities because I really do feel there is an aspect to that nature of work—whether it be air traffic control, 9-1-1 dispatch, or management of the utility grid—that we, as software engineering practitioners, can learn from and apply to make shipping software more simple, transparent, and predictable.
To that end, today, I’m launching a company: Release Engineering Approaches.
We provide a full range of build/release engineering consulting services including release engineering consulting, build engineering automation & tooling, and DevOps infrastructure support.
Our approach is to integrate those “operational” lessons and practices that so many other industries have (often painfully) learned, to make shipping whatever type of software you need to ship… as simple as possible. Every time.
If consistently and simply shipping your bits are something you struggle with, give us a shout out and let’s discuss how we can help!
To new approaches!
aviation, continuous integration, organizational interactions, processes, release engineering, strategy
Preed on Build/Release Engineering, Releases, Releng Machinery, personal | 4 Comments »
V1047
Friday, March 9th, 2012
I’m still very much enjoying Robert Glass’ The Facts and Fallacies of Software Engineering1
I’m still making my way through it, but I wanted to call out a corollary to one of the facts he covers (which even he calls out as possibly controversial):
An Australian colleague, Steve Jenkin, suggested to me his view of the rate of progress of the software profession. It’s average experience level, he said, has tended to remain constant over time. … [W]hat he meant was: with the explosion of newcomers arriving in this fast-growing profession, the increasing experience level of the growing-olders is more than overcome by the low experience level of the hordes of newbies. As I thought a bit about what Steve said, I came to this thought that I would like to introduce as a corollary:
The wisdom of the software field is not increasing.
If our experience level is not growing, then neither is our wisdom level. In fact, in our mad rush toward the new, we tend to discard much of the old. (For example, software’s newest and most popular methodologies, like Extreme and Agile, tend to make a point of rejecting the accumulated wisdom of the older methodologies.)
This insight particularly spoke to me.
It may be the local environment in which I work, or maybe the (professional) [echo?] chamber I choose to put myself in. Maybe it’s a pattern in the history that repeats every four-to-five years2, or maybe it is a fundamental shift in the industry.
In any event, the “DVCS solves all developer problems” and “Cloud solves all infrastructure problems” and “NoSQL solves all scaling problems” and “Rails is the only way to develop a website”-cacophony I often find myself in the middle of… may just be an indication that Glass’ corollary is, indeed, right.
Fred Brooks wrote almost 25 years ago that despite claims to the contrary, no silver bullet exists to reduce complexity of the software development process. And yet it’s become almost fashionable in the space to proclaim as loudly (and proudly) as one can: “Here’s a statement that implies I don’t know the history of the industry in which I practice my craft.”
If true, that’s certainly another point in support of Glass’ corollary.
I’m apparently not the only one who’s noticed the industry may be waking up to what Glass posits was true all along.
What do you think?
_______________
1 Thanks again for the reco, Ali!
2 What I tend to refer to as “an Internet generation”
agile, commentary, organizational interactions, process, strategy, teamwork
Preed on Build/Release Engineering | 4 Comments »
V1042
Thursday, September 29th, 2011
Editor’s note: original post was split into two parts; part II will get published on Monday.
I was catching up with an old college-buddy of mine, Brad1, recently and we started swapping work stories. Since he’s a software engineer, we (predictably) stumbled onto the topic of build engineering.
He was frustrated at work: the build tools weren’t what he would’ve chosen, builds took too long to build (and rebuild), adding new modules was a pain, and his build team seemed content to ignore the problems. He struggled to understand why the build engineers weren’t more helpful: “We spend the entire project lifetime working on the build tree, whereas the product ships just once, at the end,” he reasoned.
“Wouldn’t it make more sense,” he continued, “to dedicate more resources to making our lives easier? It seems like it would surely pay back in increased developer efficiency.”
I certainly know where he was coming from: one place I worked, you used to be able to get a good game of foosball in between incremental rebuilds2. And while it always killed me, I explained to Brad why it was never fixed: “If I went to a VP of Engineering and said ‘Well, we can’t ship the product today, because I spent my time making [some subset of] your developers more productive,’ he’d fire me. And he’d be totally right to do so; developers are my secondary customers.”
He retorted: “If we actually accounted for all of the time we spent rebuilding broken trees, recovering from dependency-fails, waiting for branches to get fixed, etc. etc. etc. you’d get fired too. And he’d be totally right to do so; without the developers, after all, there is no product.”
While I was slightly miffed by the sentiment, it’s nothing I hadn’t heard before. And, in the strictest interpretation, he was, of course, right.
But I thought about it some more and I realized we were both right. And we were both wrong.
Every environment I’ve worked in has entailed a mix of “developer support” and “build infrastructure construction/maintenance.” I took a hard line position to make a point3, but Brad’s expectations of the build team weren’t right either: it’s always been a continuum4,5.
If I were to plot this continuum, it’d look something like this:
Apologies in advance to the absolutely wonderful Indexed.Click to big-ify.
Some areas of note:
- Note that the Shipping Software band takes up the middle of the X-axis, but as more is invested in build infrastructure, this band will actually shrink.
- The Build Guru’s6 sole responsibility is to monitor continuous integration, work with developers to fix failures, and generally provide interrupt-driven services for developer/program managers. Usually seen in bigger teams, with big budgets.
- Crank Builders and Turners work on the “crank” that releases your software; some environments find it useful to make this distinction; the separation’s success largely depends on management and personalities involved, I’ve found.
- The area of the spectrum marked by poor decisions is where the build team has such little influence and the environment is so developer-driven that the act of shipping the bits is negatively impacted7.
- Someone who turns out to be a surprisingly good Source of Great Specs is engaged in developer support, but has the ability to influence those processes, which often gives the role on the other side of the graph a lot of help in the requirements for a build infrastructure that can directly and positively impact developer productivity.
- The “Possible Warning Zone” exists in any environment where the build team has maximal influence, but is disconnected from the developer-support function. It’s not a given: a build team focusing on requirements solicitation from the engineering organization can help provide an almost-utopian environment for developers; if they team doesn’t, it’s painful. The good news: I have never seen a build team in this position.
- The “Bad Process Feeding Zone” marks conditions where, given the correct circumstances, poor process flourishes, either by creation of new, bad process, or the stagnation of old, bad process.
- Build Engineer Burnout Central marks the zone where the organization will just burn through people left and right: mindless turning of the ineffective crank, with a focus on nothing put pushing software out the door. Good engineers will get bored and leave. Poor engineers will get stressed out. And leave.
So we have a nice graph to describe where build engineers focus their time and effort, and now we can discuss where they are, where they might move to support the organizational needs and what issues that may result as that all happens.
I’ll talk more about this X-axis, and why Brad and I were both simultaneously right and wrong on Monday.
_______________
1 That’s probably not his real name
2 Two if you were lucky or bad at the game
3 I had prefaced our conversation with this caveat
4 Which he admitted afterward as well
5 It’s one of things I actually enjoy about release engineering
6 This was a term we used at VMware; I’ve heard the role dubbed other things
7 A recent example: the complications that ensued for those switching to distributed version control systems without any real plan; someone said “This is great, we’re doing it,” and it was done
organizational interactions, people, processes
Preed on Build/Release Engineering, Releng Machinery | 2 Comments »
V878
