Continuous Integration

Release Schedule

April 2014
« Mar    



Simply ship. Every time.

At Least History is Consistent


From How Silicon Valley Became The Man:

That’s all a way of ignoring the systems that make the world possible. One example from the ‘60s that I think is pretty telling is all the road trips. The road trips are always about the heroic actions of people like Ken Kesey and Neal Cassady and their amazing automobiles, right? Never, never did it get told that those road trips were only made possible by Eisenhower’s completion of the highway system. The highway system is never in the story. It’s boring.

—Fred Turner

Apparently, well-designed, stable (even revolutionary!) infrastructure has always been boring.

As have “the squares” who toil to build it.

A Post-Planet Module World


I’ve been a Mozilla Planet module peer since the module was created in 2007.

The module, as incepted then, might have never existed, were it not for the issues I raised… but alas I’ve told that story already, and if you really care, you can click that link.

As originally created, our role was to attend to requests related to Planet’s settings and feeds, respond to technical issues as they arose, and ensure that Planet and the content it hosts serves the needs of the Mozilla Community.

Functionally, this came down to two major activities: researching and approving RSS feed additions and deletions. And making policy around content-appropriateness, including rulings whenever questionable content appeared on Planet.

We’ve been coasting on both of those mandates. For awhile now. And yet, this paralysis isn’t particularly surprising to me.

There have been instances where we, as Planet Module stewards, have been asked to “sit out” of discussions related to Planet1 and its operations. Whatever your opinions on this, it’s had the side-effect of gutting momentum on any initiatives or improvements we might have discussed.

It’s gotten so bad that there have apparently been a number of complaints about how we’ve been [not] doing been doing our jobs, but that feedback hasn’t effectively been making it to the module leadership. At least not until there are people with pitchforks.

There’s so much frustration about this issue, a Mozilla Corporation employee emailed the Planet list last week, all but demanding to be added as a peer. There was no hint in the message that any discussion would be tolerated. And while I disagree with his methods and logic, I certainly understand his frustrations.

Additions and deletions to a config file in source control aren’t rocket science.

And as for our other role—creating an acceptable-content policy for Planet and discussing those issues as they arise—Mozilla now has a Community Participation Guidelines that supersede anything a module owner or peers would say.

The fundamental issue, at the time, was one of free speech2 versus claims of “hostile work environment” and “bigotry.” While it’s unclear who is responsible for interpreting the Guidelines3, the Guidelines do take precedent over the Planet Module.

Without the policy responsibility, the Planet Module seems to me to be effectively reduced to something you could challenge a summer intern to automate4.

Who knows what the Planet Module will become. But I’m not interested in joining on that journey. So I’ve resigned my Planet module peership5.

So, as they say… so long. And thanks for all the RSS feeds.

  1. Don’t ask me by whom; I’ve asked repeatedly; I’ve been refused an answer every time
  2. Specifically, endorsing a political position
  3. Though, one can guess: I’ve only ever heard one answer to “Who is the final arbiter at Mozilla?” as long as Mozilla has existed
  4. Though, like most problems presented to software engineers, it’s not as simple as you might imagine
  5. Technically, effective last week on February 9th; but it’s been a busy week, so I didn’t have time to write about it until now

Now Showing: Your Life


My grandmother passed away last week. She was 85.

Due to a series of poor choices1, compounded by the geriatric randomness, her last few weeks were spent in a county nursing facility. So the platitudes we all hear2 about “being in a better place now” are most certainly true.

When I look back upon time spent with her, I can safely say she was one of the few3 souls I’ve known with any familiarity or emotional intimacy whom I’d considered tortured. Of course, being a teenager during the prime years of World War II4 left her with scars I can’t begin to fathom.

In the aggregate, she did the best with the skills she was afforded, in the environment she was dealt, in the cultural context of the era she landed in. And I understand and empathize with that.

Alas, she spent most of her retirement years searching for solace from those demons. Oddly repeating her first years in this country, she spent much of her retirement moving around. I think she was searching for happiness, as if it were a physical place you could pack up and move to. I’m not sure if she ever understood that many of the things that made her stressed and unhappy were within, along for every move, and getting rid of them to get to that “happy place” involved some serious self-unpacking.

That’s probably one of the lessons I’ve learned from her that I’m most grateful for5.


Experiencing the death of a family member earlier in life prompted me to think more than I ever thought I would at that age6 about what happens when we die. And I find myself thinking of it again now.

Obviously, no one really knows what happens when we cease to be of this world7.

One of my favorite ideas about the immediate-afterlife is that of a grand movie theater… one of those that reminds us of Hollywood’s Golden Age8. A concession stand has all of our favorite treats. As we sit down in the empty theater—in our favorite row—an usher comes by and gives us a remote control.

As the movie starts, we soon realize it’s the story of our lives, from birth to death. As we get past the shock of watching ourselves be birthed, the purpose of the remote becomes clear: we can fast-forward, rewind, and zoom in and out of the movie unfolding before us. Think those movies with grandiose, interconnected plots9, that weave in and out of each other. Except… y’know… you and the people you knew.

That girlfriend that broke up with you? Now you’ll know the real reason why. That company you loved, who hired a director to replace you? Watch the scenes in the VP’s office and know what really happened. That first time you drove a car, or soloed in an airplane, or really felt like you grokked that Thing You Love: experience again how awesome it was. Those months after That Person died that were a visceral blur of emotion? You can watch those again too, in an as much or little detail as you’d like, as many times as you’d like. The fast-forward, rewind, director’s commentary, and “Special Features” buttons are yours and yours alone.

I find this idea alluring because parts of the life I’ve lived, I feel like I don’t have a clue why they unfolded the way they did. And as a pattern-matching engineer, that bothers me a lot. I really wish I knew the rich, complex story of the good times, so I could repeat them and the intricacies and influences of the bad ones, so I could avoid them in the future.

There’s a twist, though: this cozy theater, with all of your favorite snacks refilling endlessly, and an almost-infinite story to re-experience… sounds a little like purgatory. And that’s sort of the point.

As I go through life, deciding what it’s OMG REALLY IMPORTANT I care about and what may be less so… as things blindside me and I react (or falter while doing so)… as I think back on all the mistakes I’ve made, I hope such a great theater, where we truly get the opportunity to understand What Really Happened exists.

Part of me also feels like the goal of life here is that once we’re offered that ticket, we’re able to politely decline it, secure and happy in the choices we’ve made and the outcomes that ensued, and having made peace with the people and events that didn’t turn out the way we had hoped and planned.

I try really hard to live like that, even though I know I wouldn’t be able to decline that ticket right now… too many unanswered questions.

Assuming it’s anything like this, I suppose none of us knows how long we’d want to stay in that dimly lit theater, replaying our story… until that moment where that Featured Presentation starts.

  1. Probably the last legally-competent ones she made
  2. And say
  3. Only?
  4. On the eventually-losing side, too
  5. It’s also one of the lessons which I struggle the most with, and which scares me the most that I won’t ever learn… by I digress
  6. My early 20s
  7. Which is part of what makes it an interesting thought exercise
  8. Not unlike the Castro Theater
  9. A la Heat, Crash, Babel, Valentines Day

The Rise of BlahOps


Last week, Perforce’s Matt Attaway and I got into an interesting twitversation de-constructing our beloved “DevOps” word.

It all started because I received an email from a recruiter with a job description referencing a “DevOps team”1, but with a description2 that sounded eerily familiar to the bread and butter of a build/release team.

Matt suggested I start a “BuildOps” movement, where I “Take all the existing text on build; s/build/BuildOps; profit!” Despite lamenting the overuse of the term “$blahOps,” the sad part is: he’s probably totally right; there probably is a mint to be made with some awk‘ing there.

What I thought most interesting, though, was his last observation on the matter:

We’re stuck with $blahOps I think. Movies have trained us to think ‘BlackOps’ are cool; anything + Ops sounds more awesome.

I’d never thought of it that way, but I think he’s onto something there. The imagery of Army-green helicopters, with soldiers jumping out of them, yelling “Go, go, go!” is not far in our imaginations whenever the word “ops” is uttered.

The irony of it all is that as many organizations try to transform themselves to the new, improved “DevOps” versions of themselves, they still ignore or under-invest in operationalizing their teams and their infrastructure. Myriad examples abound in anonymous, complain-y tweets: the CI server under the desk (or “in the cloooouuuud!” paid for with one developer’s personal credit card account3), the “just throw all the bits on S3, it’s fine” “artifact-management strategy”, the continuously muddled communication about who is responsible for which tasks, the shoving others aside justified as “just gotta get my job done”4, the indictment of the mere notion of repeatable process or consistency across actors5, the withstanding people not in right roles6. These are all antithetical behaviors of these “[Black]Ops” teams we so idolize. And yet, we think if we tack Ops on, some of that magic will imbue itself to our endeavors.

Of course, the part those movies often gloss over is the many hours7 these teams spend training and getting to know and trust each other, and how much stock they put in process (and yes, even Process) for the simple things, precisely so they can save their energy, mental and otherwise, for the complex and unexpected.

The lesson, illustrated once again is: just adding “Ops,” Dev- or any other kind, to your organization isn’t going to solve problems, business or otherwise… even if it does sound cool8.

  1. Strike one
  2. “… an opportunity for a Sr Build/Release Engineer to join a DevOps team responsible for Build, Release software packages [sic] as well as deploying to different Cloud [sic] environments.”
  3. They’ll never leave the company, obvs
  4. Instead of working with and trusting them
  5. “Because, dammit, I have thoughts and feels on Python formatting AND THEY’RE IMPORTANT SO EVERYONE NEEDS TO LISTEN!!”
  6. In effect, sabotaging the team
  7. Years?
  8. It’s not lost on me that the word “DevOps” has just the right mix of stop consonants and affricates (for English speakers, anyway) to really roll off the tongue… and I doubt it’s lost on our subconsciouses, either.

Mobile Doesn’t Suck: The Web Made Us Lazy


I recently read Tim Bray’s Software in 2014, a sort of “state-of-the-industry” post, including what to watch for in the upcoming year.

It was a good read, but I found one of his observations, on the mobile space, interesting:

The update cycles are slow. Days in the case of iOS, hours for Android (compared to seconds for browser-based apps). What’s worse is that you can’t even count on people accepting the mobile-app updates you send them. Got a critical data-losing account-compromising privacy-infringing bug? Sucks to be you.

I LOL’d when I read this. Literally. Read More

The Data-driven Dilemma


Data-driven decision making. Since appearing on the tech scene about a decade ago1, it’s seemingly become all the rage in Valley management culture.

It’s a pillar of the “lean startup” ethos. It’s alliterative. And it’s arguably become the dominant decision making paradigm, trumping even time-honored, classical engineering analysis, for established tech companies and startups alike.

It’s an alluring proposition: it purports to remove ego from discussions, making the data you bring to the table (and can explain with analysis!) the first class citizen, not your title. It tries to open up the paths of communication, so the flow of decision making isn’t top-down, allowing ideas to flow both ways. Lots of research today, especially in the lean manufacturing space, indicates that the people on “the factory floor” are make decisions that result in better outcomes than managers reading reports in another building and dictating policy and process from there.

After years of employing the strategy, some have been able to “adapt” the method to re-establish the patterns data-driven decision making was supposed to address. I’ve seen this firsthand, but what I found interesting is these (anti-)patterns had been observed by others too.

Some interesting examples of the contortion of the data-driven mantra:

Read More

  1. From what I can tell, anyway; if I didn’t know better, I’d say it seemed like Stanford had a data-driven decision making course it required all its MBA and engineering management masters students to take

Skip the Betas


When I saw an ad1 for Betas, Amazon’s original comedy series about a group of Silicon Valley millenials building the next disruptive dating app, my curiosity was piqued.

But after watching the first two episodes2, I have to say: Betas is completely unwatchable.

The first two episodes cover what you’d generally expect: character exposition, contrived situations serving to bring our protagonists together and establish our antagonists, the awkward setting of social/sexual tension and, of course, setting the story arc for the season: the trials and tribulations of building of a mobile application in a startup incubator.

Unfortunately, the execution is all too contrived: the characters, from the shy, quiet Indian wunder-engineer to the pretty, upstart early 20-something “CEO” who has just a touch of “Aspies” to the jamming-with-Moby venture capitalist aren’t particularly innovative. The “comedic” situations our stereotypes find themselves in aren’t inspired—posing as Larry Paige to crash a VC’s house party? Really?—nor is the character interplay or “tension”: at times, I feel like I’m watching an episode of Glee, except without the singing.

Or the diversity.

If, like me, you were hoping a show like this might tackle that particular issue our industry finds itself grappling with3, you’d be correct, though probably not from the angle you might have hoped.

Betas ticks all of the boxes off that checklist:


We’ve got free broad band, free snacks, and not to mention: those bangin’ ass hotties with the bargain hunting app. [Camera pans to a group of five women around a table; only one of them has a laptop.]

(Because obviously, women entrepreneurs’ most noteworthy trait is their “ass-banging’ness.”)

Oh, and the antagonist-team’s rival app? “Valet-me”: from the makers of “Tit-stare.”

Racism, check:

I’ve got a Bitcoin on Bollywood.

Uttered as the character raises a physical coin (Haha, get it? Currency!), referring to her Indian—red dot, not feather, a riff we hear in the same episode—teammate.

Drinking culture? Our VC has you covered there; his new incubator class is informed that at the kickoff party:

Remember: Intoxication is mandatory.

Everyone claps, of course.

Oh, and let us not forget ageism:

You know I’m 35; it’s like 95 in Valley Years…

Now I know this is a sitcom, and the point is to get a laugh. A lot of things were said that… were sort of questionable, but I didn’t bother noting them because other sitcoms have made equally tasteless jokes for an (equally) cheap laugh.

The most surprising thing about watching Betas is that I realized I was actually becoming frustrated and angry while watching it. It took me awhile to figure out why: Betas’ characters do a really good job of re-creating archetypes of our industry, so much so that they remind me of people I’ve met. But the problem is… none of those people are likable.

Co-creator Josh Stoddard said: “We were looking at this culture, and it is pretty easy to parody.” But this show really isn’t a parody, at least not in the sense IT Crowd or Hackers were. Those had characters that maybe frustrated us, but they were all likable in some way or another because they possessed certain aspects of ourselves that we could identify with.

I don’t identify with any of these characters. Nor do I want to. Heck, I wouldn’t even want to have coffee with them.

The other frustrating part is I couldn’t really figure out who this show was supposed to be for; I came up with two possibilities:

Read More

  1. In print; on a Muni bus shelter no less!
  2. A small sample size, yes, but Amazon offers the first three episodes free, and then you have to pay; so they have to hook you in three
  3. What group has Paul Graham decreed incapable of programming today?

Old Checksums Collide Hard


Our industry still insists on using MD5 checksums. And I don’t know why.

Since 2004, it’s been widely reported that “the security of the MD5 hash function is severely compromised.” When talking about collision attacks in hash functions, the answer to whether or not they’re broken—and thus useless for the sole purpose we employ them—is not unlike answering whether or not one is pregnant.

And yet, I still see MD5 checksums floating around all over the place, providing a false sense of security.

It’s interesting to speculate on why this might be.

Obviously, there’s the inertia argument: release infrastructure is one of the least exciting places for the business to invest in, so if you’re publishing checksums at all1, changing the hash functions—”What do those even do? And does any customer use them?”—probably isn’t the top of management’s list.

(It doesn’t help that OS vendors don’t make this easier either: OS X Mavericks still ships an command line md5 tool, but nothing to easily calculate SHA12. And you wanted to easily checksum a file on Windows? Good luck with that.)

Perhaps it has to do with the fact that (oddly?) most people just don’t seem to care to validate that the bits running on their machines are what the software company intended to ship? I guess they assume that nobody is tampering with their Internet traffic? It’s obviously a point of consternation when their machine goes rogue on them, but… that never actually happens, right?

Anyway, ever since 2006, I’ve been suggesting to my clients that they use SHA1 checksums, and forgo even publishing MD5. But that advice may not be correct anymore.

At the recent Chaos Computer Club conference in Germany, researcher Rüdiger Weis presented evidence (English translation) that SHA1 should be taken as effectively broken too.

The news is particularly interesting, especially in light of the recent revelations regarding the NSA: part of his justification for arguing its compromised status is that SHA1 is in the same family as SHA03, which was developed by the NSA, without a published technical specification; it, too, is considered broken. He also notes the modification to SHA1 was made by the NSA without any technical explanation.

(Weis puts it more colorfully: “I would personally rather put a wager on Britney Spears’ virginity than as to the security of SHA-1.”4)

His recommendation for a hash function? SHA35. Unfortunately, it’s less clear what we should do.

Security guru Bruce Schneier says SHA3 isn’t necessary, and SHA512, a member of the SHA2 family, is as of 2012 “still looking good6.” It doesn’t help that even if we wanted to use SHA3, I couldn’t find a utility on Linux implementing it; sha512sum is implemented in coreutils 8.x.

Maybe we’ll all end up moving toward what Gentoo does, and brute-forcing it by validating multiple checksums, from multiple families; portage currently checks: sha256, sha512, Whirlpool and file size.

Either way, anyone care to wager on how long it’ll take for us to stop relying on SHA1 checksums to validate bits? (That is… if we ever do…)

  1. Good on you!
  2. Yes, I know you can use openssl for that purpose, but it’s still annoying
  3. And SHA2, for that matter
  4. Translation mine, with the help of Google Translate
  5. For encryption, he recommended against ECC, in lieu of 256 bit AES, 4096 RSA DHE and use a 512 bit hash function
  6. I guess he’s less worried about NSA-backdooring than Weis
OS/2 Warped

A WARP’d View of History


Ars is carrying an interesting historical article on the history of IBM’s OS/2. The article describes it somewhat provocatively (though, probably equally realistically) as “half an operating system.” Definitely worth the read if you’re into classic operating systems or computing history.

I knew most of the information, largely from Robert Cringely’s PBS series Triumph of the Nerds1, but one thing I didn’t know about IBM’s repeated marketing foibles with the product:

OS/2 version 3.0 would also come with a new name, and unlike codenames in the past, IBM decided to put it right on the box. It was to be called OS/2 Warp. Warp stood for “warp speed,” and this was meant to evoke power and velocity. Unfortunately, IBM’s famous lawyers were asleep on the job and forgot to run this by Paramount, owners of the Star Trek license. It turns out that IBM would need permission to simulate even a generic “jump to warp speed” on advertising for a consumer product, and Paramount wouldn’t give it. IBM was in a quandary. The name was already public, and the company couldn’t use Warp in any sense related to spaceships. IBM had to settle for the more classic meaning of Warp—something bent or twisted. This, needless to say, isn’t exactly the impression you want to give for a new product. At the launch of OS/2 Warp in 1994, Patrick Stewart was supposed to be the master of ceremonies, but he backed down and IBM was forced to settle for Voyager captain Kate Mulgrew.2

I guess this explains OS/2 Warp’s funky logo.

(The article also includes an interesting ad wherein an aging (and “somewhat befuddled”) cast of M*A*S*H is inexplicably surrounded by a bunch of IBM PC boxes.)

Apparently Reimer does a lot of articles on historical OS’s; you should check out his Ars publication list if that’s your cup of tea.

  1. Which author Jeremy Reimer specifically notes that he took much of the information from and “[view] multiple times during research.”
  2. I don’t know why they’re giving panning Captain Janeway; she’s a great jailed Russian mobster
Older Posts