Eulogy for a Founding Father

05/13/2013

About a month ago, I noticed a tweet from Coop:

Pouring out a little liquor for tinderbox today. Drinking the rest, because, you know, tinderbox.

It linked to a mozilla.dev.planning post describing the plan to end-of-life Tinderbox¹.

As one of a handful of people who was required in an employment-capacity to support Tinderbox in production^2,3, I can certainly understand the elation at getting rid of the aged continuous integration system. It hasn’t changed much (or seen much maintenance for that matter) since its original open source release fifteen years ago and certainly had plenty of warts⁴.

Having said that, part of me is sad at the… glee, for lack of a better word, at its demise.

Tinderbox is certainly antiquated by any modern standard, but it should not be forgotten that, having been released in 1998, it is very much the grandfather of continuous integration systems.

It may have “sucked,” but it facilitated a workflow and nurtured an ethos that is not only extremely important, but taken for granted today: namely the notion that individual developers should be “on the hook” when checking in, and have a responsibility to their peers to monitor the build and make sure the tree “stays green.”

It was Tinderbox that was largely responsible for introducing a generation of software engineers to this now-commonplace concept, and helping to get a previous generation of engineers to care about such things. Mozilla was the poster-child user for Tinderbox, but I know of at least VMware and Yahoo who used it years before Hudson/Jenkins and Buildbot existed.

Beyond that, it sports features that those systems TO THIS DAY do not:

1. Tinderbox put its build logic in the clients, and had them report to the server via email; this may seem odd now, but the asynchronous nature of that data flow meant that Tinderbox was surprisingly tolerant to network failures, something Jenkins and (especially) Buildbot both continue to handle horribly.
2. Tinderbox supported JSON output⁵ that allowed the development of its successor, tbpl, and other tools; it was one of the first CI systems to make its collection of data consumable in such a transparent format.
3. As mentioned above, Tinderbox was modular, separating out the logic of the client from the server, and using a simple API to communicate between them. This meant it was trivial to write a Tinderbox client in whatever language you preferred to write one in, as long as it output something the Tinderbox server expected; this is huge, and a major distinction between today’s systems, which follow a command-and-control model. I personally know Jenkins users who tell horror stories about 40+ minute startup times when tens-of-thousands of clients were involved. Tinderbox, by design, never had this problem.
4. Tinderbox facilitated continuous integration for open source projects in a way that still has yet to be replicated; say what you will about using email as a communication mechanism, but it allowed outside parties to set up and maintain “weird” platforms corporate sponsors of an open source project didn’t feel like investing in⁶, and yet still provide those CI results to the community in a highly-visible, inclusive way⁷. Good luck getting those Jenkins and Buildbot ports open through the corporate firewall today. And for all the hype about Git’s decentralization, there isn’t a continuous integration tool today built around a decentralized model like Tinderbox was.

It’s easy to hate on tools like Tinderbox when you purposefully ignore the context in which they were created⁸: Richard Stallman’s venerable GCC compiler was “a pile of crap”… if you forget it was the most widely used compiler before the EGCS fork, over a decade before “Clang” was more than a Klingon General⁹. “CVS sucks!” … yeah, compared to Git, Subversion, and Perforce. But not compared to RCS or Microsoft Source Safe. “Shell scripts are old-school garbage!”… except if you were trying to automate something in the 80s and early 90s, and didn’t want to have to learn about C pointers to do it.

It’s unfortunate that it seems unlikely Tinderbox will ever get the respect it truly deserves as one of the founding fathers of this idea of continuous integration that most of us take for granted today.

But perhaps that’s merely another example of Alan Kay’s indictment of our industry for its “disdain of history.”

In any event, to paraphrase Apollo 13 Mission control: “Farewell Tinderbox… and we thank you.”

_______________
¹ And the requisite bug, which you can conveniently find find by searching for keyword “tinderbox-death”↵
² Coop also holds this distinction↵
³ And as one who also has a number of (now embarrassing) checkins into that code base↵
⁴ Including some pretty bad ones↵
⁵ And possibly others↵
⁶ Firefox on AIX, anyone?↵
⁷ It’s telling that the last few projects still using Mozilla’s Tinderbox instance fall pretty squarely into this category: Camino and Bugzilla↵
⁸ Especially when they’re (merely) broadly used for fifteen years, but not actively maintained↵
⁹ Not really↵