Continuous Integration

Release Schedule

May 2020
S M T W T F S
« Nov    
 12
3456789
10111213141516
17181920212223
24252627282930
31  

Branches

Twitter

Simply ship. Every time.

The Elevator Storyteller

08/30/2011

I recently read FastCompany’s interview with John Lilly.

I knew John from my time at Mozilla1, so it was interesting to hear him talk about the developments after my tenure at the Corporation.

One thing he said stood out to me:

[W]hen I was at Mozilla the activity I did mostly was to tell the story—tell the story simply, understandably over and over and over again.

… I’d feel bad about telling the same story 50 times, so I started to change it and embellish it from time to time. What I realized is that if I told a different story to 50 different people, then suddenly the whole organization will be slightly out of alignment. And so, my job was to tell the story in a simple way that was repeatable and amplifiable, not to make it all diverge.

This was an interesting insight to me.

I have struggled2 to explain precisely what I do and why it’s important.

I’ve had varying degrees of success over the years, and the success I have had has been inconsistent.

As I thought about the above excerpt, I realized what I may have been missing: a simple, coherent, engaging way to accurately, yet consistently and succinctly tie it all together: the so-called “elevator pitch.”

***

I was phone screening a candidate recently, and they likened their role as a release engineer to that of a mailman: “She doesn’t write the letter; she didn’t drop it in the mail; she didn’t make the envelope; she’s not going to read it, but she’s critical to moving the contents around and ensuring it gets delivered.”

While I liked the analogy, longtime readers know I prefer a different one: air traffic controllers.

I’m often asked why I continue to find this analogy so relevant.

  • Air traffic control is operational in nature: functionally, it involves a lot of planning and talking, but it’s fundamentally an exercise in scaling process to keep people safe and keep things moving.
  • There’s a level of detail orientation that many other professions simply do not require.
  • It’s an open secret that air traffic control towers are often understaffed and under-resourced; this can turn situations that would be small problems into catastrophes, which leads me to…
  • No one pays attention to air traffic controllers until catastrophic events occur, and then (and usually only then) are they under the spotlight.
  • Air traffic control is a service-oriented role, but to maintain sanity and ensure consistent outcomes in a complex environment with many actors with competing requirements and goals, they must do so within a framework3.
  • It’s well-known that it’s a high stress environment4.
  • Controllers’ (advertised) responsibility is the safe, expeditious movement of traffic, but they must prioritize that necessity with procedure and air craft emergencies and deviations; good ones know when and how to safely issue shortcut instructions.
  • On its face, controllers’ work seems simplistic: “Move the blips around, make sure they don’t hit each other and talk on the radio all day? A chimp (or maybe a computer) could do that!” This prompts some interesting perceptions about their role5.
  • A pilot won’t realize why air traffic controllers are useful if they only fly on sunny days with no turbulence in the middle of nowhere, by themselves.
  • Controllers can be one of your best friends when something goes wrong with the plane and a pilot finds themselves declaring an emergency.

Replace “controller” with “build engineer,” “traffic” with “release” or “check-in” and “plane” with “project” and you’ll start to see why I find the similarities so compelling, at least when release engineering is approached the way I approach6 it. Now, to turn it into a narrative like Pushing Tin.

Or maybe Top Gun.

_______________
1 I have more-than-a-few-vivid memories of interactions with him, including when he interviewed me
2 Especially back at Mozilla
3 And expect pilots to do the same
4 [Never ending] Firedrills anyone?
5 And in many ways, informs how they’re treated
6 Pun intended

Deckin’ Gentoo into Tweetin’

08/23/2011

Editor’s note: if you don’t plan on installing TweetDeck on your Gentoo box anytime soon, you can probably skip this post…

Ever since I’d started using Twitter, Echofon or the Twitter website itself had sufficed for my needs.

But recently, I found myself using Twitter in different ways—separate accounts, mostly1—than I had in the past and needed a client with a few more features.

Echofon’s releases have continued to slide in the quality department ever since they started wrapping their API keys in weird native XPCOM libraries, and on Linux at least, there’s been an annoying bug for the last few releases where the popup background is totally transparent, and takes 3-5 seconds to paint, making it impossible to read any text2.

I know some people who swore by TweetDeck though, so I decided to give it a try on my Mac at work a few weeks ago. I’ve found it to be a very capable client for what I’m starting to need.

I decided to try installing it on my (Gentoo3) Linux laptop, so I could use it at home.

After some Googling, I found some instructions from 2008 for doing exactly this.

The instructions worked… mostly; I needed to make the following minor changes:

  1. As you might expect, both AIR and TweetDeck have changed since then; one major problem is Adobe keeps moving the link for the AIR SDK around (which you need, because they apparently don’t ship just-the-runtime on Linux [anymore?]). This is the current link: http://www.adobe.com/products/air/tools/sdk/thankyou/linux.html.
  2. When you get to that page, you need to have JS on so that the auto-download starts; in some awesome release engineering failure, the link-to-the-thing in the “If it does not start, download it here”-text IS DIFFERENT (and won’t work with the 0.38.1 version of TweetDeck) than what you get if the download JS fires4.
  3. There is one minor error on the instruction page which will trip you up: it says to launch the app with: /opt/AIR-SDK/bin/adl -nodebug /opt/AIR-apps/application-name/META-INF/AIR/application.xml /opt/application-name; directory naming preferences aside, this is wrong. The second argument should be the same as the directory-portion of the first argument, i.e. /opt/AIR-apps/application-name in this example.
  4. If you get the error message “Invalid application descriptor: descriptor version does not match runtime version” that’s because you downloaded the link-the-”OMG DOWNLOAD HERE”-text links to, which won’t work5
  5. When I got TweetDeck launched, it came up (looked as pretty as the Mac, too!), but immediately started complaining about it “having trouble using some of your passwords that are stored securely on your machine. Clicking Submit6 will clear this data so that you continue to use TweetDeck”; it’s nice enough to send you to a FAQ page, but this page talks about deleting your Gnome keyring and all sorts of other draconian solutions; read the ENTIRE PAGE before following any of its instructions.

    In my case, it turned out that since I’m using XFCE (but still have all the Gnome goop installed), Air couldn’t figure out that it wanted to use Gnome7; you need to set GNOME_DESKTOP_SESSION_ID=1 in the environment, and it will work fine.

Other than that, everything has seemingly worked OK!

the only (notable, admittedly) problem that I’ve run into is when I click URLs that aren’t something TweetDeck loads internally, nothing happens; I have to copy/paste.

That may be something with my default web browser in the Gnome goop not being entirely set, though.

I’m looking forward finding all the various bits that kept everyone raving about TweetDeck!

_______________
1 @SoberBuildEng for professional stuff, @preed for personal stuff, and the always-popular @ShitPreedHears for… “stuff”
2 Let’s not discuss the bit about how these types of Firefox extensions aren’t the type Mozilla cares to support anymore, cf. rapid release, and say we did…
3 No ricer jokes!
4 Which, for the record, is a 32 meg file: 95af29afa9b01d0b75d430f854f1c3a061f3fbf7 AdobeAIRSDK.tbz2
5 It’s an old version of the SDK that isn’t compatible with the current version of TweetDeck
6 Which is confusing, because there’s only an “OK” button…
7 It actually told me this in the terminal: Unknown desktop manager, only Gnome and KDE are supported

Nightly’s on First; Aurora’s on Second

08/18/2011

I wasn’t going to even bother saying anything, but raccettura’s post goaded me into it.


Let’s put this in some perspective:
Apple—user experience and design queen Apple—is to the right
of Mozilla’s position on this issue!

Allow me to succinctly cut through all the cacophony on this: version numbers matter1.

They’ve always mattered. And they will continue to matter.

Why?

Because the first question anyone asks when debugging their own software problem or helping someone else: “What version are you using?”

“Latest” (or the even more asinine “infinite!”) is not even in the universe of a useful answer (and there exist a universe of reasons why it may not even be an accurate one, either).

Version numbers help users find the appropriate release notes for the application sitting in front of them.

Version numbers help users search forums and knowledge bases, and be able to correlate others experiencing a similar problem and use the correct instructions for the version they have.

Version numbers make it possible for savvier users to help themselves, by knowing what to put into the bug database when they search.

Version numbers make it possible for users to help developers, being able to effectively communicate when reporting a bug.

Version numbers are what the press refers to, so they can communicate which feature set their review pertains to.

Version numbers are used by organizations and startups to drive PR and user buzz about new features and product families: “We just released version 2.0! You should really check it out!”

Software labeling—version numbers, as consumers know it—has been a core of communication involving the answer to “Which actual bits are running on my CPU?” since… well the beginning of packaged-and-sold software.

Anyone quoting3 “Nobody asks ‘Which version of Facebook are you running?’” is easily dismissible: it’s a great example of the fallacy of false comparison, but it’s not worth responding to4,5.

In Mozilla’s specific case, this versioning consternation induced by their “rapid release” schedule is a direct result of versioning-as-a-concept being a proxy for something—API compatibility—in the Mozilla platform itself.

It’s important to call this out, because it’s necessary to distinguish between “rapid release causes my extensions to break every six weeks”—to users’ annoyance—and the (non-response) response of “Well, version numbers don’t matter anymore.”6

I see the move to obscure the version number to really be about reducing user “upgrade friction” by making the process as opaque and secretive7 as possible. This makes it easier to use Firefox’s user base as a lever to “move the web,” an argument I’ve previously made. This move is a consistent and logical step toward doing just that.

An interesting thought exercise to consider: what would the response be if Microsoft started updating Visual Studio on random intervals, and refused to divulge the version number installed on your machine when you tried to find it. Given that Mozilla bills the Open Web as a platform, and Firefox the tool to interact with and develop against that platform, it’s not a dissimilar example.

In any event, simply saying version numbers don’t matter anymore, and repeating it over and over like a bird you’d feed crackers to, does not suddenly make it a fact of modern software development.

It illustrates a profound lack of understanding of versioning’s (important) function in the best case.

And disingenuousness in the worst.

_______________
1 You’d probably expect a release engineer to say that, given that version numbers and versioning as a process development area has always been a large part of my job2
2 But that’s not why I’m saying it, entirely anyway
3 I should say “repeating the sentiment,” because I can’t find any Google record of anyone (publicly) having said that; the only reference is back to Laura Thomson’s original post, where the quotation is uncited
4 But I will: software which I download and run on my computer is very different than a web application
5 Facebook, Google Apps, and lots of other websites deploy software that’s broken all the time; people generally don’t have to answer “What version is it?” because, there often isn’t a way to meaningfully interact with those companies about issues in their (web-based) applications; you just wait until it’s fixed
6 In fact, this entire discussion about removing the version number from the UI is probably more acrimonious than it otherwise would be, because Mozilla has not, to date, resolved the issues raised by those negatively impacted by their “rapid [fire] release” decision
7 User experience people would probably choose “unobtrusive”

“One More Thing…”

08/15/2011

An OH:

When you make your users feel like this, you’re doing it wrong.

When you’re driving your users toward something they don’t want such that they just go grab the “original” version of what they don’t want, you’re doing it wrong.

And when you’re telling your users they’re wrong for wanting what they want, you’re doing it wrong.

I think that sums it up perfectly1.

I can’t help but wonder if this new tide of product management methodology is a reaction to Apple’s focus on product design in general and Steve Jobs’ reputation for making those product decisions in specific, and the success it has brought AAPL.

Problem is: Steve Jobses are rare2 and Apple spent decades surrounding itself with customers who are interested in joining the Church of Steve3.

The saying used to go “The customer is always right.” With millions of users, obviously it would be difficult to provide the exact product or service that every customer wants.

At this point, I’d settle for merely moving away from “The customer is too stupid to even have a comprehension of what they want, so we don’t even have to listen to them anymore.”

_______________
1 Mozilla is the example du jour, but is certainly not alone
2 Duh, but apparently this needs to be said
3 And the Cult of Mac

Doc Review: Mindsight

08/13/2011

When Mindsight was recommended to me, I immediately bought it.

It arrived a couple of days later, and promptly sat on my bookshelf—through a move—for months.

I just wasn’t ready to read it.

But times and circumstances change1 and, as with so many things, I find myself asking “Why didn’t I read this sooner?!”

Mindsight is Daniel Siegal’s newest book2. His writing style is both conversational and engaging, even though he often refers to complicated brain chemistries and structures (You’ll never look the same way at your enclosed fist again). His patient histories provide the lively characters for a book whose major plot is a scientific exploration of the “squishier” parts of our most important organ.

The question at hand: how can we understand and experience our consciousness as an interested, yet compassionate third party—a concept Siegal dubs “mindsight”—to achieve what he calls integration3 within our minds.

Mindsight provides medical research to back up the technique, which interestingly is what I’d describe as a form of guided super-meditation; as an engineer, I appreciate the scientific digressions discussing why meditation has been humanity’s go-to for psychological serenity across both centuries and cultures.

My only disappointment was that the book doesn’t get as far into the details of Siegal’s techniques as I would have liked. He does talk at a very high level about what has worked for his patients, but leaves just enough out that trying it yourself can be a bit daunting. It doesn’t help that his patient case studies make it all look so easy4.

To be fair, Siegal doesn’t intentionally mislead his readers; from the epilogue:

This proposal is no easy task. Dissolving fixed mental perceptions created along the brain’s firing patterns and reinforced relationally within our cultural practices is no simple accomplishment. Our relationships engrain our early perceptual patterns, deepening the ways we come to see the world and believe our inner narrative. Without an internal education that teaches us to pause and reflect, we may tend to live on automatic and succumb to these cultural and cortical influences that push us toward isolation.

Maybe it’s someone whose conflict resolution process involves stabbing her co-workers in the back, figuring they’ll never compare wounds with each other. And she can’t figure out why she’s not the success she thinks she should be.

Maybe it’s an acquaintance who proudly boasts about his dating flavor-of-the-week. In between shots at the bar.

Maybe it’s the guy who became angrily exasperated with his friends when they couldn’t sooth the grief from my father’s death. And couldn’t figure out why they became frustrated and left

We all know people similar to these, either out about in the world or within the confines of your own consciousness. Mindsight provides insights into the tools to start figuring out the most important question in addressing them: why.

(If you’re in a hurry, or just curious what this is all about, you can listen to Siegal’s Google Tech Talk on Mindsight.)

_______________
1 And bookshelves
2 One of many
3 Siegal discuses eight “domains of integration”: consciousness, horizontal, vertical, memory, narrative, state, interpersonal, and temporal
4 They’re all so happy at the end of the chapter! And always cured!

Not Everything is the Circus

08/09/2011

It was the quintessential Twit-versationTM.

Mike Shaver tweeted a couple of weeks ago “you can make a surprising amount of stuff happen just by making people believe it’ll happen, and that it’ll be fun to make it so.”

I replied—admittedly sarcastically—that he, “clearly [had] never been a build engineer.”

Shaver corrected me, but didn’t address the point I was implying. And then it devolved into a critique of analogies, never again to return to the discussion.

Fin.

Thing is, even though my reply was sarcastic, I really did want to substantively discuss it; it’s a topic I struggle with constantly1.

Let us be honest: the core principles and tasks involved with configuration management often provide a roadblock to software development. Anyone saying otherwise is deluding themselves2.

Learning some new tool, merely to submit code you’ve already written to some magical file-store? That’s annoying. Even having to check in3 puts a speed bump in developers’ cognitive flow.

Monitoring some rainbow-colored waterfall of build information after you check in? More distracting annoyances. Plus, it takes4 time.

Coordinating and documenting changes to the official build environment instead of just making them? “Why do I have to do this?”

And yet, all of these are (just a few of the) bread-and-butter tasks for a well-run build/release organization, and they’re all widely regarded as best practices for modern software development.

Obviously, the friction on developers’ work flows and on development teams themselves created by these activities can be reduced5. But they can never be totally eliminated. And none of those tasks are particularly “fun” for developers.

Just like scrubbing the bathroom toilet isn’t “fun.”

Or chemotherapy: not “fun.”

Getting permits for that remodeling project? Wouldn’t describe that as “fun.”7

Doing the fourth release in two days from the Christmas dinner table: no fun.

But they’re sometimes necessary. And we do them because the alternative is worse.

Unfortunately, motivational-speaker pablum won’t change the fact that they’re necessary, nor address the lack-of-fun involved.

I sincerely would love to get people genuinely excited about Perforce integration procedures or release environment reference specifications or build system tools8, so that developers didn’t think of all of these things as such chores.

I would love it if it were as easy as parroting that these things are fun, but as someone who’s been in this role for almost a decade now9, I can tell you it’s never been that simple. It has always relied more on a combination of pain/risk avoidance, management support, and resource allocation and less on “motivational mantras.”

That was my point.

Of course, to the prospect of finding new mantras to try when “No really, this will be fun!” doesn’t work, I say: I’m all ears.

_______________
1 I had gotten distracted from writing this post, to the point I hadn’t finished it; when I ran into exactly this issue again this week, I decided to…
2 Or trying to sell something
3 Sorry, sorry, “push”
4 “Wastes”
5 Especially when the release teams serving them aren’t resource constrained6
6 A niche I find myself having a knack for randomly continuing to fall into…
7 Turns out: my roommate probably wouldn’t either…
8 Ok, maybe not GNU Autotools
9 And who’s still doing it

TeamAndrew: Anything They Can’t Do?

07/27/2011

It all started a number of months ago when @dolske randomly linked to an innocuous video about fluffy unicorns.

Little did I know the first time I watched it, but I’d end up spending hours upon hours listening to Pantaloons’ videos, and would eventually be led to his current collaborator, Mr. Gunnarolla, only to spend another host of hours watching his great material.

When they announced they were going on tour, I jumped at the thought of being able to watch them perform the songs live that had been bouncing around in my head on repeat for the last few months.

I dragged my roommate along since I had caught him singing the delightful earworms that are their tunes; we got to the San Francisco venue—a cute cafe in the Outer Sunset—slightly before showtime, but it was already so crowded we spent the entire concert by the door.

I actually saw Gunnarolla outside looking for Pants1, saving the space for him to park in; I surprised myself when I leaned to my roommate and said “OMG THAT’S HIM!” He surprised me when he went over to Andrew and said “Hi! We’re big fans. He got me started watching you!”

Andrew said “Aww, thanks guys, that’s great!” And then surprised us both with a hug.

Unlike other shows2, we were treated to an opening set by Sam Hart (which had been announced in a really catchy way). Sam’s songs were beautiful ballads to the things he loved: loves-that-weren’t, loves-that-were, his cat, and MarioKart.

Then the Andrews took to the stage, and… how can I put it? Singing. Contests. Singalongs. Awesomeness. Just like their videos… except, y’know, not on YouTube. And we were in the videos.

My day started with a corrupted source control server and went downhill from there. But it ended with this:



Team Andrew finds the sauerkraut in my lederhosen after the San Francisco concert

I also nabbed one of their awesome tour posters and got them to sign it!

It started out a lil’ rocky, but all things considered, it was a spectacular day.

I can’t wait to see them on tour again, hopefully soon!

_______________
1 Wait, that… sounds wrong…
2 I think

Bootstrapping Quick Releases

07/21/2011

I just released a new build/release harness1 I’ve been working on for about a year now called QuickRelease.

How does this relate to the Mozilla community, you might ask?


Some bootstraps are clearly more…
ornate & complex than others…

Well, some of the core ideas behind QuickRelease actually come from an automation project Rob Helmer and I worked on back in the day called Bootstrap; you can still find it if you know where to look.

Bootstrap had a lot of good ideas2, but it failed for a few reasons: its core design was based on some object-oriented patterns, and since the implementation language was Perl, that was all sorts of pain3.

Additionally, Mozilla, like many organizations at the time, was moving to Python for much of its infrastructure tools; Perl was considered gauche. Combine this with the “omg new hotness”-push in 2007 to use the buildbot-hammer to smash every problem4, and Bootstrap’s demise was pretty much assured.

I never thought hitching everything release engineering-related to the buildbot wagon was a particularly well and thoroughly conceived idea. In the ensuing time, it looks others have discovered the precise point I was making back then. There’s been some work on adding a layer in between buildbot and the build system… which is exactly what Bootstrap was.

And what QuickRelease is.

It’s worth noting QuickRelease is in production use, shipping others’ software, including some Mozilla-based projects.

Go give it a gander; let us know what you think.

_______________
1 I didn’t push the initial announcement to Planet
2 And in fact, one of QuickRelease’s best ideas—unit testable steps—was rhelmer’s…
3 You have to be pretty blessed in the head to do any hardcore OO-Perl…
4 Using unwieldy master.cfgs

Hatin’ on the Critics

07/19/2011

Allow me say this as plainly as I can: all “haters” are critics1, but not all critics are “haters.”

During a recent (mostly-online) discussion, one of the responses to the issue at hand that I (indirectly) ran across was “don’t be a hater yo.”

It’s a common response in tech circles these days, especially among open source projects and startups in pursuit of high visibility and using 3rd party developer-engagement to do so.

The statement is many things, but it is not, I repeat, not a substantive response to a reasoned question.

In fact, it’s even worse than a substance-lacking response: it frosts the non-response response with a thin, sugary layer of ad hominem frosting.

Every time you say it, you might as well be saying “I don’t want to be burdened by having to explain my position; I want to continue thinking what I think and I don’t want anyone or any thing2 to challenge my opinion. So shut the fuck up before that happens3.”

Which is a totally fine response to give to someone who’s challenging you.

But don’t delude yourself that muttering something about “haters” is the same as making a cogent point or somehow adding substance to the conversation.

When you Godwin-ily decry “the haters,” it’s an indication that you’ve failed to do either, and you’re hoping no one else noticed.

But it turns out: we do.

_______________
1 Mmmm… probably
2 “Data!”
3 And people—”geeks,” actually—are confused/surprised/angry at the suggestion that there’s a new new geek anti-intellectualism growing?

“It All Ends Here.”

07/17/2011

I recently had to re-download Firefox1.

I’m not exactly sure what new features Firefox 5 sports, but based on the landing page, it must have something to do with Harry Potter:

Now I’m a little slow.

Mozilla is “reinventing itself.” Got it.

But can someone help me understand what a half-page about the new Harry Potter movie has to do with Mozilla’s Mission2?

Or the Open Web?

Maybe Dumbledore knows.

_______________
1 I put myself on the beta channel some time ago; I wanted to help test security fixes before they were rolled out to users; but with the new and improved versioning scheme, that’s not what “beta channel” means anymore.
2 Maybe I mean “Mozilla’s Manifesto” here?3
3 I’m not sure which one I really mean…4
4 See? Little slow.

Newer Posts
Older Posts