Reading [your post], I couldn’t figure out what replaced Tinderbox for the Mozilla builds. What feeds tbpl? Does Mozilla not use Tinderbox to build continuously?

When I left Mozilla in 2007, there was a Release Engineering project in progress to actively replace Tinderbox (Client) with buildbot. So in short, no, Mozilla does not use Tinderbox Client to drive its continuous integration builds, and hasn’t for some time.

Do they still use buildbot today?

I didn’t know the answer to that question, so I tracked down Coop on IRC, who graciously gave me a few minutes of his time to answer exactly that.

He said:

• Mozilla currently uses “95% buildbot, with 5% Jenkins for random small projects”
• There are multiple buildbot masters that drive the buildbot clients
• Unlike the out-of-the-box buildbot master setup, the masters query a job scheduling database instead of monitoring source control for changes themselves; they then report their results to a database, which tbpl (and other services) use to generate their reports/dashboards; the buildbot master waterfall pages aren’t accessible to the external world (which makes sense, because they include unsecured administrative functionality¹)
• There are about 60 masters right now, but Coop said “number keeps growing though, so we need to rethink the whole solution”

So there’s your answer, Carsten!

_______________
¹ A long standing criticism of mine, among others↵

For episode 19 of The Ship Show, we did a joint podcast with the Food Fight Show Crew, and had a ton of fun; if you’re looking for a high level review of the conference, that’s a good place to get a high level overview¹.

Even though it wound down over a week ago, it cracked open a world view that I’ve not had a lot of experience with to date, and that experience was pretty impactful, so I wanted to discuss it a bit more after having digested things.

It’s interesting to me that most of the people attending ChefConf are approaching many of the same topics we do in traditional “release engineering” for “native”/”boxed” software, but it’s similarly patently obvious that the dynamics and interactions² can be pretty different too.

To whit: I found it very interesting that among a sea of developers in talks and in the hallways, I could pick out common themes that wouldn’t surprise any practitioner: version control, deployment, scaling, configuration management… but one phrase I did not hear uttered once by a single person was “release engineering.”

What is especially interesting was having heard bits of pieces of various conversations, it’s clear that the problem space people were tackling had a great deal over overlap with the issues release engineering has been tackling for years.

It’s just no one called it that.

I haven’t quite been able to figure out why that is.

It’s possible that “release engineering” may have a “curmudgeony” connotation to it, so there’s a desire to distance from (any) roles that have traditionally been at odds with “lean manufacturing”³ concepts. Or, perhaps, it’s just that the bulk of the attendees at ChefConf hail from a so-called “[Web]Ops” background, and release engineering (and release engineers) weren’t an obvious part of the Web 1.0+-world?

I don’t know… but no matter your opinion of release engineering, all y’alls are doing it⁵… even if you don’t know it, or find the concept abhorrent.

Which brings me to the next thing I found fascinating: we did a word association segment for The Ship Show with various conference-goers that turned out not only be a lot of fun, but revealed the wide, often extremely disparate viewpoints on some standard ideas within the DevOps sphere. For instance:

• “Release engineering” prompted responses spanning from “confusing”, “tiring”, and “hopefully a thing of the past” to “sanity”, “frictionless”, “really cool”, and “craft.”
• “Configuration management” prompted “hard” and “Jeezum… yah… no… I dunno” all the way to “good”, “sanity”, and “important”⁷.
• “Shell scripts” engendered, on the one hand “old school”, “horrifying”, “obsolete”, “the fallback”, and “if you have to” all the way to “kinda awesome sometimes”, “I like ‘em”, “much needed”, and “actually pretty awesome.”
• Our beloved “DevOps” excited utterances ranging from “I don’t know”, “ouch”, “dear God”, “confusion”, and “not-a-Thing” to “sweet”, “cool”, “so cool”, and “fun.”

You can draw your own conclusions⁸, but as someone who spends his days thinking about DevOps, the people who identify as part of that community, and the problems it faces, I found this illustrates the group’s real heterogeneity.

In attempting to fix broken social and technical systems, I think we can often forget or gloss over the fact that we’re, and the systems we work on, aren’t as uniform as we might think, which is probably something we should all be more aware of.

In the podcast “revue,” we talk at length about the keynote presentations; but the bulk of any conference is the small talks, and the breadth of topics at ChefConf was particularly surprising. There were plenty of technical talks to be sure, but in keeping with Opscode (and the Chef community’s) focus on culture, there were plenty of “soft skill talks” and those turned out to be among my favorite genre of the “short talks”:

• Joan Touzet gave a very post-modern talk on “Coming to Terms with Chef” and her own journey as an “old-school sysadmin” to a Chef-convert; what I liked most about this talk is that it’s applicable to anyone struggling with change in their industry and learning new tools. is being forced to learn new tools.
• Jeff Hackert gave a talk I can’t imagine giving, wherein he basically admitted to be an asshole in the past and went on to describe how he noticed that, came to terms with it, and now helps coach others to not-be
• The sharing of mistakes wasn’t specific to just cultural “soft subjects”; both Julian Dunn and Sascha Bates gave great talks on how to avoid failure as a new help sous-chefs⁹

(While it doesn’t even challenge the experience of attending the conference, Opscode has graciously put most of the talks online; not only is this great if you couldn’t attend, it takes some of the sting off about attendees missing talks they really wanted to see!)

One other topic I found thematic in a lot of the (especially keynote) talks was the focus on the global economic changes, and how this relates to technology (broadly) and IT, ops, and software development specifically. Adam Jacobs touches on the core issue in his keynote with the discussion on Ubernana.

Hearing that idea from numerous speakers, it really revealed how fundamental the changes in the industry are.

I kept thinking of tectonic plates: they feel like they’re not moving, but if you look at them from space¹⁰, you realize they’re floating on top of molten rock flows, which very much move. The shift in those plates is starting to become more palpable.

It is interesting ponder where those lava flows will take the tectonic plates on which our industry is built in this next year.

And I’m sure we’ll discuss it at ChefConf 2014.

_______________
¹ Plus, hear me having a bit of a potty mouth↵
² Both systemic and sociological↵
³ What DevOps is really about⁴↵
⁴ So I’m told…↵
⁵ And in some cases, reinventing the wheel…⁶↵
⁶ Which makes me sad↵
⁷ Along, of course, with the winner for the most responses: “Chef”↵
⁸ if there are any to be drawn↵
⁹ I originally used that term in episode 19; can I claim to have coined it now?↵
¹⁰ They discuss this specifically at 51:30 in↵

Pouring out a little liquor for tinderbox today. Drinking the rest, because, you know, tinderbox.

It linked to a mozilla.dev.planning post describing the plan to end-of-life Tinderbox¹.

As one of a handful of people who was required in an employment-capacity to support Tinderbox in production^2,3, I can certainly understand the elation at getting rid of the aged continuous integration system. It hasn’t changed much (or seen much maintenance for that matter) since its original open source release fifteen years ago and certainly had plenty of warts⁴.

Having said that, part of me is sad at the… glee, for lack of a better word, at its demise.

Tinderbox is certainly antiquated by any modern standard, but it should not be forgotten that, having been released in 1998, it is very much the grandfather of continuous integration systems.

It may have “sucked,” but it facilitated a workflow and nurtured an ethos that is not only extremely important, but taken for granted today: namely the notion that individual developers should be “on the hook” when checking in, and have a responsibility to their peers to monitor the build and make sure the tree “stays green.”

It was Tinderbox that was largely responsible for introducing a generation of software engineers to this now-commonplace concept, and helping to get a previous generation of engineers to care about such things. Mozilla was the poster-child user for Tinderbox, but I know of at least VMware and Yahoo who used it years before Hudson/Jenkins and Buildbot existed.

Beyond that, it sports features that those systems TO THIS DAY do not:

1. Tinderbox put its build logic in the clients, and had them report to the server via email; this may seem odd now, but the asynchronous nature of that data flow meant that Tinderbox was surprisingly tolerant to network failures, something Jenkins and (especially) Buildbot both continue to handle horribly.
2. Tinderbox supported JSON output⁵ that allowed the development of its successor, tbpl, and other tools; it was one of the first CI systems to make its collection of data consumable in such a transparent format.
3. As mentioned above, Tinderbox was modular, separating out the logic of the client from the server, and using a simple API to communicate between them. This meant it was trivial to write a Tinderbox client in whatever language you preferred to write one in, as long as it output something the Tinderbox server expected; this is huge, and a major distinction between today’s systems, which follow a command-and-control model. I personally know Jenkins users who tell horror stories about 40+ minute startup times when tens-of-thousands of clients were involved. Tinderbox, by design, never had this problem.
4. Tinderbox facilitated continuous integration for open source projects in a way that still has yet to be replicated; say what you will about using email as a communication mechanism, but it allowed outside parties to set up and maintain “weird” platforms corporate sponsors of an open source project didn’t feel like investing in⁶, and yet still provide those CI results to the community in a highly-visible, inclusive way⁷. Good luck getting those Jenkins and Buildbot ports open through the corporate firewall today. And for all the hype about Git’s decentralization, there isn’t a continuous integration tool today built around a decentralized model like Tinderbox was.

It’s easy to hate on tools like Tinderbox when you purposefully ignore the context in which they were created⁸: Richard Stallman’s venerable GCC compiler was “a pile of crap”… if you forget it was the most widely used compiler before the EGCS fork, over a decade before “Clang” was more than a Klingon General⁹. “CVS sucks!” … yeah, compared to Git, Subversion, and Perforce. But not compared to RCS or Microsoft Source Safe. “Shell scripts are old-school garbage!”… except if you were trying to automate something in the 80s and early 90s, and didn’t want to have to learn about C pointers to do it.

It’s unfortunate that it seems unlikely Tinderbox will ever get the respect it truly deserves as one of the founding fathers of this idea of continuous integration that most of us take for granted today.

But perhaps that’s merely another example of Alan Kay’s indictment of our industry for its “disdain of history.”

In any event, to paraphrase Apollo 13 Mission control: “Farewell Tinderbox… and we thank you.”

_______________
¹ And the requisite bug, which you can conveniently find find by searching for keyword “tinderbox-death”↵
² Coop also holds this distinction↵
³ And as one who also has a number of (now embarrassing) checkins into that code base↵
⁴ Including some pretty bad ones↵
⁵ And possibly others↵
⁶ Firefox on AIX, anyone?↵
⁷ It’s telling that the last few projects still using Mozilla’s Tinderbox instance fall pretty squarely into this category: Camino and Bugzilla↵
⁸ Especially when they’re (merely) broadly used for fifteen years, but not actively maintained↵
⁹ Not really↵

What follows is an excerpt from the first one:

A family friend picked me up and we started the trek back towards Fort Collins. We heard one last update as we sped off from KDEN that they had him out of surgery and were waiting to revive him. He said the doctor quoted her a “10% chance,” but then was honest with him and said “about a 1% chance.” I don’t know why they bother trying to quantify it below 50%; at that point, it’s all the same: “We don’t really know.” Or “We don’t really want to tell you.”

It was at this point that I finally got the full story: Dad had been packing things up with Mom this morning when he fell to the floor, complaining of pains in his leg; he thought he had pinched a nerve in his back. They waited a few minutes and then Mom asked if she should call an ambulance. To her surprise, he said yes.

Upon arriving at the hospital, an EKG was done to normal results. Next, a chest X-ray series was done, but he moved as X-rays pierced his body, requiring them to re-shoot them. They decided to do a CT scan instead. They rolled him over to radiology to get the series done. From what I understand, as soon as the tech finished the series, he noticed the tear in the aortic valve. Just as he went to call the doctors to get him into surgery immediately, Dad arrested on the table.

In front of Mom.

She was quickly pushed out of the way and they began working on him, trying to stabilize him. It didn’t work, so they put him on an artificial heart pump, and took him up surgery to repair the valve. He never woke up.

The doctor declared it all over an hour or so after the three hour surgery when they tried to warm his body back up and get his heart pumping again. In repairing the valve, they had won the battle… at the expense of the war.

By the time I got there, Mom had already been told. [Kennan] had gotten there, heard, and consoled her all while I was still on the highway, trying to get here.

I remember walking into the “conference room”, a mini-waiting room, really, and seeing [Kennan] and one of Mom’s friends surrounding her, her muffled wailing snapping me right back to when I first talked to her on the phone. She had mentioned that she had dyed her hair blond for a wedding a couple of weeks ago… but she looked older and more tired than I remember her, and at first glance, it looked like her hair had spontaneously turned a yellowish shade of gray… but gray to be sure. I sat down next to her and held her. I didn’t say anything.

Save for a family friend hugging me immediately upon seeing me, no one actually clarified that he had passed. I remember Mom saying something, and I asked “Is he here in the ICU, or?” and Mom said “He died,” spurring off more wailing. I felt so stupid for not figuring out what had happened.

Everyone left the conference room and it was just Mom and I for a few minutes. I didn’t say anything, save “He’s fine now” when Mom uttered “He was in so much pain” while fumbling with Kleenex. Again, I felt stupid not only for what I said, but for what I didn’t… or rather, not knowing what to say.

A “comfort counselor” then came in and introduced herself. She said she was sorry—something I’d hear numerous times again that night. Mom continued crying, and I must have had this really stoic, uncaring look about me; I remember staring off into space at nothing in particular, every muscle in my face as tense as it could be. Not a tear willing to show itself. The chaplain came soon after and said more sorries. I began to wonder if it was hard for the comfort counselor to do her job; I would think it would be difficult to muster up the energy to say all those sorries all the time and really mean it. I guess I was angry at the time… maybe because I felt like they were making me look stupid since I didn’t have anything meaningful to say.

Mom asked if she could see Dad one last time; they agreed, but warned her that he would still have the surgical tubes protruding… a requirement until the autopsy was completed, which we later found out it wouldn’t be… the cause of death was obvious. The surgical duty nurse came back and said they were ready. Mom lead the way… (seemingly) strong as always. I followed, and everyone came in trail of me.

I can’t convey to you what it was like to stand outside the hospital curtain in the surgical ICU, listening to your mother cry in pain and just have to stand there and let it happen. After about five minutes, she beckoned us all in, and I again just stood next to her, hugging her, and remaining dumbfoundedly silent and “unemotionally” stoic.

His eyes were closed. There were stains of dried blood on the white, plastic body bag they had already put him in… and smears of pink on the cotton blankets. There was a large tube sticking out of his mouth, taped to his cheeks and accompanied by a smaller tube jutting out of his nose, tied off in a knot encrusted with a dull, rust colored blood.

They had cooled his body down for the surgery while he was on the heart pump, his body was artificially cold to the touch. Mom was holding his hand and I put my hand over both of theirs. And just stood there.

Mom got a cell call that was going to take a few minutes, so I asked to go back into the ER to have a few minutes alone with Dad. I went back into the empty surgery ICU and pulled the curtain behind me.

I squeezed his limp hand and looked at his peaceful face.

And began to cry.

In some ways, tech is a much smaller and more incestuous scene than I ever imagined – and I grew up in the midwest. There are also people in it who are dishonest, manipulative, abusive, bullying, mean-spirited, harassing and destructive. Early in my career I was very paranoid about maintaining amicable relationships with these individuals or staying quiet despite my moral qualms about their actions, because I was always told I’d have to work with them again, and that someday they might be on the other side of a hiring board or committee or collective I needed something from. I’ve since realized that these very fears ensure these assholes will have long prosperous careers, where we’re all forced to see them again.

Definitely worth the read.

I get asked a lot of times why this blog tends to rely heavily on footnotes; it turns out it’s mostly a historical footnote².

The original incarnation of this blog started back when I was one of Mozilla Corporation’s two³ full-time release engineers. Since a lot of the community communication regarding builds came through that blog at the time, a lot of the funnier content⁴ had to be relegated down to the footnotes.

And then it just sort of snowballed from there; it turns out people found them amusing and missed them; they’re also a great tool for citations, without being too intrusive.

So yeah… they’re here to stay… for now.

_______________
¹ Thanks to Redfive for calling it out!↵
² See what I did there?↵
³ And then only one… and then two again↵
⁴ Or stuff that were inside jokes anyway↵

He had a lot of interesting stories and data about introducing PaaS infrastructure at Mozilla to service web developers’ infrastructure needs. If you’re interested in how Mozilla’s forays into PaaS have fared, definitely check the episode out.

A great listen, and a huge shout out to Brandon for taking the time to join us.

And, “if build engineering, DevOps, release management, and everything in between” is of interest to you, feel free to join The Ship Show Crew every couple of weeks for discussions on precisely those sets of topics!

_______________
¹ Yes, the episode title—PaaS: Play or Paasé—is a little cheesy…^2
2 But I picked it, so I guess I only have myself to blame…

When Google finally released their maps app, I decided it was worth trying iOS 6. The new Do Not Disturb feature sounded intriguing and well… there’s no other path to security updates.

But my experience since the update has been… less than stellar:

• Cell reception in the house—which, to be fair, has never been awesome the two years I’ve lived here, but has been usable—has become non-existent.
• Battery drain up through 6.1.1 has been at least 20% more; the phone can lose as much as 20% just sitting on the night stand for a couple of hours. Admittedly, this problem probably isn’t helped by…
• I have to reboot the phone anywhere from 6-10 times a day; through some yet-to-be-determined usage pattern, it will get into a state where it will claim three or more bars of signal, but no texts will send or receive and calls to my Google Voice number, which ring both the cell and a land-line simultaneously, no longer reach the iPhone. The only thing I’ve found to fix this problem is reboot.
• Probably worse: the phone silently “freezes” when it gets into this state… so I often don’t realize I’m missing texts or calls until the landline rings or people ping me on GChat/Skype to ask why I’m “not f*@#$!ing responding to [them]!!” The phone thinks it has 3+ bars though!

Out in the world, I haven’t noticed reception to be better (or worse¹) than previously, but the phone’s iOS 6 induced schizophrenia doesn’t inspire any confidence. And how would I even know I was missing the calls and texts²?

The whole situation is discouraging to say the least. But as a class of consumers, we’ve signaled to vendors for quite awhile now that we’re willing accept low quality in products and service.

What’s most frustrating about this is:

• There is no real remediation process. I should be able to downgrade my phone to a stable, last-known working version of its OS³; but there’s no way to easily do this. And if I do it, then there’s no security story for that⁵
• Apple really seems to be mishandling their iPhone business lately. I’m not sure why this is⁶, but when you have carriers texting their customers to wave them off of your software updates, there’s a serious problem somewhere. And we don’t need to delve into the iOS 6 Maps app debacle. All in all, I don’t know what consumers are supposed to make of it, other than two big companies pointing fingers at each other while they’re unable to use the product and services they’re paying monthly for… with potentially life threatening consequences⁷.
• Apple and AT&T reps need to stop promulgating the myth that iOS updates are incapable of affecting reception. I have a CS degree, so I understand the techno-babble customer service reps spew⁸. But please: just stop it. Immaterial of the technical reason, people—numerous people—are unable to use their phones in the ways they flawlessly had for months previous to the update. Yes, the data may be anecdotal on a customer-to-customer basis, but you’re not fooling anyone, especially when you read Apple’s own update messages:
  
  

  Yeah… keep workin’ at it, gang…

I updated my phone again a couple of days ago, and the problems seemed better for the first few hours⁹. Lamentably, though, my phone is back to imitating that mostly-useless brick now… and I’m back to waiting for Apple to ship an update that unbreaks-for-realsies their non-iPhone 5 customers.

Because iOS 6.1.1 sure ain’t it.

(And if you have an iPhone 4(S) that you have yet to upgrade: save yourself!)

_______________
¹ I do live in San Francisco after all…
² Another lovely symptom of this problem: AT&T will batch 2, 3, 5-hour old texts for me: when the phone reboots, if I’m lucky, in the few minutes before it goes signal-deaf, I will get a stream of texts that has prompted my roommate to ask if I have Hector Salamanca hidden in my room…
³ Before you ask, yes: I ran a factory reset at home; I was dismayed to learn that it doesn’t actually reset the phone to the state it was in when I got it from the factory, i.e. with iOS 5^4
4 Also to my dismay, it blew away all my apps, including my carefully never-upgraded 3.x Twitter client
⁵ Because mobile-WebKit has no bugs, right? Right.
⁶ Enter the “Steve Jobs would never do this…”-chorus
⁷ The somewhat-obvious, but seldom discussed pink elephant in the room…
⁸ “The modem firmware is unchanged by the update, so it’s impossible for OS updates to cause these problems… they’re totally separate systems…”
⁹ Maybe that’s just because anything was better than iOS 6.0

Hashtag-sadface.

While I enjoy writing, I’ve also been distracted by working on producing The Ship Show, a podcast we describe as covering “Release engineering, DevOps, and everything in between.”

My co-hosts are really awesome, and we’re working on episode 8 right now. Some of our more popular shows include:

• Is there such a thing as “too much automation?”
• An interview with CFEngine’s Mark Burgess
• Hiring the Best DevOps Engineers

And our last show, on bootstrapping developer workstations, includes the commentary I meant to write about Sascha Bates’ blog post, on “being a therapist.”

So, instead of reading it, grab that episode, and start listening at about 45 minutes in.

I suspect more and more posts-I’d-write-here will end up as content on the podcast, so if you enjoy the Sober Build Engineer, you should definitely subscribe to the podcast.

Is Your DevOps Instrument Rated?

Thursday, October 4th, 2012
7:00 pm – 7:45 pm
61 Wall Street Atrium

More Details: http://luny.org/events/is-your-devops-instrument-rated/

and after:

Ship Show Happy Hour!

8:00 pm – until we settle which is better: Python or Ruby
Killarney Rose
127 Pearl Street, New York, NY

More details: http://luny.org/events/the-ship-show-meetup-with-j-paul-reed/

Feel free to join for either or both!