Continuous Integration

Release Schedule

May 2020
« Nov    



Simply ship. Every time.

The Cost of Unpaid Experience


By now, most of you have probably heard about local news station KTVU’s gaffe involving the names of the Asiana 214 pilots. If not, you can read all about it and watch the cringe-worthy video.

There is no question that the practical joke falls somewhere between poor taste and racist1; for some reason, this particular crash has brought out that side of the media.

The part I find interesting about this specific gaffe is that the NTSB did in fact confirm these names… so it wasn’t a prankster in the newsroom, which is what I had first assumed2. As the story evolved, we found out that it was, in fact, an intern who relayed this information to KTVU.

A couple of points stand out to me about this whole fiasco: Read More

  1. Though, truth be told, I will admit: I did LOL when I first saw it… though largely out of shock as much as anything else…
  2. Having worked in a newsroom, I have stories about “jokes” accidentally making their way through editorial and getting published, to similarly… “poor outcomes”

EPISODE 21: Going, Going... Gone.


After the Going, Went


Episode 21 of The Ship Show (“Going, Going… Gone.“) is over a month old, but with Crazy JuneTM1, I never made it around to discussing the episode. Plus, there were some interesting developments that took some time to percolate.

If you didn’t hear the episode, the Crew discussed the various considerations and issues that are worth taking into account when deciding whether or not to stay at a job. It was an interesting episode to tape, because it’s often hard to have that conversation and entirely remove the identifying information.

In fact, it was such a delicate topic that we ended up putting this episode in the can for a few months, just in case.

If you’re an employee and didn’t have a chance to hear it, I’d definitely take the half-hour and listen; it’s turned out to be one of our better discussion/advice episodes to date.

The one thing I was a little sad about it because none of us come from a management background (really), we didn’t really have anyone with the managerial perspective on this issue. I think it would have been really interesting to explore

  1. As a manager, have you ever lost someone good? Did you see it coming, or was it a total surprise?
  2. How did you handle it? Did you try to keep them? Were you empowered by the organization to offer them anything substantive? Were you successful?
  3. Do you wish the person had talked to you before making a decision? What is your advice for reports on the best way to start that conversation?
  4. Have you ever had a report leave, and after hearing their reasons, think to yourself: “they’re totally right?” What did you end up doing then?
  5. Why do you feel the it’s so difficult to have these conversations? Is there a way we could improve upon that, from upper-management to middle-management to line engineers, without it being… weird?

We may have to redo the episode at some point, and get some management input on these questions… I think that’d be very valuable.

In any event, even though the topic was a delicate one, I personally received a lot of positive feedback.

No less than four people took the time through various channels to to tell me that this episode prompted them to really re-think their work situations.

Two of those announced their new jobs a couple of weeks after the episode shipped.

  1. Mostly related to getting my #DevOpsDays talk ready!

Armchair Airplane Analysts


In the wake of the Asiana 214 accident, it’s been interesting to observe the differences in how the government media response, mainly through the NTSB, has changed since the last large-scale aviation accident in the United States, the Colgan 3407 crash.

In short: the NTSB, largely via its Twitter feed, has been communicating directly with the public about the facts they’re finding, seemingly as they discover them. This is in stark contrast to previous investigations, where information was more curated and polished and released in batches on a schedule. And none of it was considered “finalized” until the full report came out, a process which can take years to complete.

There is little question which model the public1 prefers.

But I do worry that the end result is not the more-informed public we’d assume. In fact, if the coverage of the Asiana 214 incident is any indication , we’re seeing more rampant2 speculation, not less.

In this situation, the NTSB has done an incredible job of presenting the facts in a near realtime manner to the media and the public at large; however, you’ll notice that they’ve refrained from much, if any, judgment3.

This is not the case with the media or the public.

Some anecdotes in point: Read More

  1. Especially us Millennials!
  2. And potentially dangerous
  3. I went on a long treatise on the distinction between findings of fact and findings of judgment in episode 18 of The Ship Show, starting at about 31 minutes in; check it out if you’ve never heard the distinction before.

Redesigning the Pipeline


Both continuous delivery devotees and those who enjoyed my DevOpsDays Silicon Valley 2013 talk may find this tidbit I saw in Wired a few months ago interesting: Key to Eliminating U.S. Flight Delays? Redesign the Sky Over New York City.

It’s an in-depth look at Mitre’s work with the FAA to redesign the airspace around New York City, which handles a ton of air traffic and has a number of unique requirements.

I found various elements of the project really interesting (and applicable to our industry):
Read More


Bye Bye Birdie


Songbird, the first “actual OMG real startup” I ever worked for, announced earlier this month that it’s closing its doors.

I remember exactly when I first heard about Songbird: it was at a brown-bag lunch event at Mozilla Corporation1. Rob Lord and a couple of ‘birders were demoing the 0.4 (or so) version of the desktop client and it was amazing to see XULRunner used for such a different application than a web browser.

At the time, I didn’t have an inkling I’d work there, but early in 2008, there I was, the company’s first release engineer.

Located at Howard and 2nd, Songbird was the quintessential San Francisco SOMA startup. The Nest was a beautiful office and lunch often involved going to the Irish Bank for a beer and bangers and mash.

At the time, I still lived in Mountain View, and Songbird was directly responsible for me eventually moving up to “the City.” (After about six weeks of doing the three-hour-per-day CalTrain commute, I was over it, and decided I had to move up to the Big Time.)

The economic problems of late-2008 changed a lot of things, but Songbird had built a great core engineering team, and I have many fond memories of working through those uncertain months, poking fun at the less than-stellar office space2, and working through many releases in solidarity.

I’m also proud of the hoops we were able to make XULRunner jump through. The team was great at modifying and working with the on-again-off-again3 platform to achieve something totally new. Because of our reliance on so many open source projects4, there were a lot of interesting release engineering problems to solve. Back then, we were tracking upstream open source projects, plus our own patch sets using Subversion. We were also supporting huge partners, and we learned a lot about release engineering coordination with large organizations.

For me, Songbird’s legacy is the group of people the experience introduced me to and that I was privileged to work with. The startup environment is a harrowing one, and that combined with the waxing and waning fortunes of the company created a tightly knit group of people who sometimes disagreed on engineering approaches, but always respected each other.

We still hang out in IRC together and chat on a daily basis. Every few months, we’ll get together for beer or a barbecue. We relive some of the old memories, but talk about the new things in our lives as well. I think the fact that we still do this is a testament to the type of team Songbird assembled.

It’s sad to see the Bird migrate away forever, but the effects of its song will be heard for a long time.

  1. I was still working there at the time
  2. The Nest ended up moving, but stayed in SOMA
  3. Mozilla never really decided what to do with XULRunner; to my knowledge, they still haven’t
  4. XULRunner, gstreamer, taglib, etc.; I think when we counted all of them, there were something like 15 packages

Peak tech?


Apple will never again come out with a product as transformative as the iPhone. Google will never build anything more useful than its existing search engine… . And Facebook … won’t ever be anything more than a place we share photos and links.

Interesting arguments… and I think I agree.

AAPL’s Fortunes


Like many techies in the Bay Area, I spent yesterday morning watching the keynote at Apple’s World Wide Developer Conference.

Colin Barrett graciously (and brilliantly) hosted an event at his place bringing together a group of Apple developers from all walks and featuring coffee, pastries, mimosas, and plenty of snarky comebacks for the various announcements and demos in the keynote.

It was a great way to spend the morning.

Personally, I was a little disappointed by the announcements; there was a lot of good stuff in there, but as someone who doesn’t develop iOS apps (or primarily in Mac OS at all), there wasn’t much for me specifically to drool over. (Though, as an iPhone user, some of the iOS 7 announcements did interest me, especially Control Center… and no, I don’t care if they’re copying Windows Phone.)

AAPL, June 10th, 2013.

I was later looking through the iOS 6 stock apps to compare the icons to the new iOS 7 ones I happened to randomly launch the Stocks app; AAPL popped up, and I noticed a couple of interesting things:

  • AAPL hit its daily high a few minutes (ten, maybe?) into the keynote
  • It’s hard to tell exactly, but there seem to be three dips during the keynote; it’d be interesting to know if these correlated to the transitions from the Mac hardware to Mac OS X to iOS announcements
  • It’s interesting see AAPL’s biggest slide for the day—leaving it down over its open—seems to be approximately when it was clear there actually wasn’t “one more thing”1
  • It looks like the day’s volume was mostly driven right after the end of the keynote

None of this is particularly insightful, though with all the purported high-frequency trading going on, it does make one wonder if these were driven entirely by the keynote announcements, or the ups and downs were amplified by trading algorithms.

In any event, special thanks to Colin2 for hosting the event; it was a lot of fun!

In slightly-related Apple keynote news, a friend of mine, Daniel Lu, posted renderings, based on what was presented at the keynote, of his guess on the guts of the new Mac Pro, plus some historical Mac cube-like product comparisons; definitely worth checking out.

  1. I attended Colin’s event with Sam Sidler, who noted “I don’t think Tim Cook is a ‘one more thing’-kind of a guy…”
  2. Who, coincidentally, starts at Apple on Monday, on the UIKit team; extra-congrats!

The Dark Ages That Never Were


I generally don’t discuss previous work experiences with the details unredacted—”Names changed to protect the innocent and/or guilty,” and all that—but Mozilla prides itself on its openness and transparency1, and they’ve been asking for #webstories, so here’s mine:

Preed & Moz

A very young, very naive me, in the lobby of Netscape’s Building 23.

My beginnings with Mozilla and, perhaps ironically, with release engineering started when I was 18. I worked for a summer on Netscape’s Build/Release Engineering team, and fell in love with the product, the culture, and release engineering itself.

To say that it shaped my career is not an understatement.

When I went off to college, I remained a part of Mozilla’s burgeoning community, working on various aspects of the project. So when I joined Mozilla Corporation in 2005, it felt a lot like returning home.

In many ways, it was the wild wild west at that time; I did my best to do the core work of any release engineer—shipping the damn bits—while balancing that with forward-looking initiatives. For a period, I was the only full-time release engineer.

Mozilla Corporation’s current Director of Release Engineering, John O’Duinn, has been presenting Release Engineering as Force Multiplier2.”

In it, O’Duinn makes some bold claims and based on reactions, audiences seem pretty horrified about Mozilla’s release engineering past.

Release Engineering in those days, like any team in any organization, wasn’t perfect by any stretch of the imagination; it was a role that had been treated a bit like hot potato, handed around from engineer to engineer since the Netscape days, and it is accurate to say it hadn’t really been invested in.

But O’Duinn does not need to make false statements to make this point.

Read More

Risky Business


While tidying up for the weekend, I got pulled into an interesting twitversation with John Allspaw and Dmitriy Samovskiy on the topic of risk.

Allspaw had linked to Johan Bergström‘s article titled “What is the risk That Amazon Will Go Down (Again)?

It’s an interesting read for those in the [web]ops space. But don’t let the title deceive you: Bergtröm was really opining on what the valid definition of risk is in our industry, and comparing it with other industries’ definitions.

The answer is a little murky (and the topic of his Velocity keynote?), but what I find more interesting is that the definition of risk is a topic of discussion.

This may at first seem a bit weird (bordering on absurd, even?), but a colleague relayed a story to me about her experience on differing definitions of risk and how it affected the work she was doing.

Read More

Newer Posts
Older Posts