One of the main features the release sports is “totally silent updates,” following Chrome’s path of “web-based version numbers”¹ and going out of the way to obscure this information.

It will be interesting to see how this plays itself out.

Firefox 10 took us into the land of “always compatible” extensions². Now with the obscuration of the act of versions changing, I wonder if the end-user’s experience will start to tend towards a more randomly-broken web browser, and users being confused why the extensions they were using yesterday are suddenly just-not-working today.

These are problems Chrome seemingly never faced, because they don’t have the rich ecosystem of addons that Mozilla does. This isn’t a particularly insightful revelation and I’ve even discussed it previously. I guess I’m still talking about it because I’ve never understood why Mozilla Corporation did it, beyond “because Chrome did it³.”

In any event, I recently got thinking about versioning again because of a conversation I had with a friend last week about version numbers and mobile.

We were chatting about our phones, and I mentioned I recently upgraded to iOS 5.1, having waited a bit to see how the update shook out on my friend’s iDevices.

He replied he’d been waiting for his carrier to roll out updates to “Ice Cream Sandwich.” I asked my friend—a college-level computer science instructor with a master’s degree in the subject—what version that was, and how it was different from… the previous version.

He answered “Oh… I don’t know. 2.3? 2.4?”

(It’s 4.0.x, actually.)

Thinking back to the features I use that were changed and updated in iOS 5.1, I said “Oh. Well… what’s different in the new release? What are you looking forward to?”

“Oh… … I have no idea. It gets stuck less in your teeth than Honey⁴ and is more satisfying on a cool day?”

The whole conversation was yet another great illustration of the point: not only do version numbers continue to matter… they matter in a marketing context.

They make it easier for your users to talk about your product in meaningful ways.

Obviously, number schemes are easier to test for equality and ordering, but they also can communicate the magnitude of change: an update from iOS 4.x to 5 communicates at a very intuitive cognitive level that the change is bigger than an update from iOS 5.0 to 5.1⁵

These are important aspects when users want to identify, discuss, compare, and contrast features, stability, bugs, and overall usefulness of the products they use and (hopefully!) love.

After it was clear that our conversation about his Android phone simply didn’t have the adjectives necessary to meaningfully discuss, we moved on.

But my confusion remains: I still don’t understand the push (in certain circles?) to make our users’ conversations more difficult than they need to be.

_______________
¹ aka “no version numbers”
² Even if, y’know, the extensions are actually totally unusable on the new version
³ I can hear my mother saying “If Chrome jumped off a bridge…”
⁴ I think he meant Honeycomb
⁵ Some may point out that these numbers have sometimes been contorted to serve a business end; it is true this happens all the time; that’s a problem that solves itself when customers get burned by it too much.

This release focuses on a request we’ve had from many people: more documentation and examples.

0.14 sports:

• Full epydoc-style documentation for QuickRelease (generated docs are available, so you don’t have to do it yourself)
• A example set of processes and steps that uses all of the various QuickRelease concepts by downloading Firefox’s source code, building it on Linux and copying the generated installer to /tmp.
• Some (hopefully minor) API changes to convert the classes to using more-modern/Pythonesque properties, where it made sense to do so.

We didn’t get to generated tarballs for releases in this release; that will be in 0.15. For now, you can grab a tarball from github as always.

We hope the documentation helps! If not, hit up the QuickRelease Google Group, and we’ll see how we can help!

Now, on to that happy hour!

It’s a great series of posts, and definitely worth your time, especially if you find you and your friends pondering the Old Guard’s tactical responses to current trends in our industry at cocktail parties⁵.

Cringely references a quotation from an old Steve Jobs interview to explain why the problem exists, and it really resonated:

The reason IBM can’t deliver is also explained well by Steve Jobs. It’s IBM’s maniacal fixation on process, once a strength but now a cancer.

“Companies get confused,” Jobs told me. “When they start getting bigger they want to replicate their initial success. And a lot of them think well somehow there is some magic in the process of how that success was created so they start to try to institutionalize process across the company. And before very long people get very confused that the process is the content⁶. And that’s ultimately the downfall of IBM. IBM has the best process people in the world. They just forgot about the content.”

This is a very astute observation⁷ and still an incredibly common problem, even today, with respect to process. You see it in organizations of all sizes, from start-ups to, well, IBM.

I think the issue stems from the notion that if it is assumed any good process can be scaled to any size organization or problem, then the structure of the process itself must be able to be extrapolated and specified. And because of this requirement that we do the heaving lifting to extrapolate, and all the work to specify it, it will remain consistent, and thus remain effective.

It’s a tantalizing notion. If true, it would mean one can make their finely tuned, tweaked, “perfect”-process into an asset, which they can apply everywhere and/or sell.

While there might be some truth in that—certainly, many attempt to make a living doing exactly that—ultimately the structure-of-the-structure of the process⁸ is more important than the structure of the process.

Put another way: good organizational (build/release) process has aspects that are clearly consistent across organizational size and disciplines, but trying to bolt the steps of a enterprise corporate security firm’s release process onto, say, a desktop client for a cloud-based media product, is a recipe for a lot of consternation⁹.

Pre-landing checklists obviously make sense for A380s just as much as they do for 182s; good checklists in both cases will have a natural flow which is backed up by the accompanying equipment’s ergonomics, seamlessly incorporate multiple checks for critical items, and are designed to fail-as-safely-as-possible.

But they’re very different checklists.

The desire to force “wrong-sized” process is what makes everyone roll their eyes and make that “I’m gonna barf”-motion capital-P Process comes up.

Figuring out what needs to go on their own release checklists, making it logically flow between their infrastructure and processes, and fine-tuning is one of my favorite exercises with my clients¹⁰.

One size certainly does not fit all, and transplanting process wholesale seldom works. But when we dissect effective process (and process management) from other disciplines, by taking a look at its structure, not the process itself, we can learn something applicable to release engineering.

And very valuable, too.

_______________
¹ Part 1
² Part 2
³ Part 3
⁴ Part 4
⁵ As I often do…
⁶ Cringely later clarifies: “In this instance content means the deliverable, whether a product or service.”
⁷ As most of Jobs’ are…
⁸ How very meta!
⁹ This may be obvious, but I see it all the time. Usually in reverse
¹⁰ If it’s something you struggle with, we should chat!

He argues that “Designers and front-end developers get all the credit,” despite the fact that the rest of the engineers supporting that whiz-bang device consumers want also make important contributions and have serious impacts on product quality and user experience.

It’s certainly a valid argument¹. It can also a very demoralizing position to be in.

One of the core components of his argument is because technical topics like “wasted CPU cycles and their effect on battery life and “the costs of (effectively) Universal binaries” can be difficult to explain to end-users, this, in-part, is why back-end-engineers remain invisible.

To these things, I can only say: “come here and give us a hug! No, no… group-hug!”

In many organizations, those performing fundamental engineering support functions—QA, release engineering, and DevOps/TechOps—know exactly this feeling, and struggle every day to make sure what they do is visible without it being in such a manner that is associated with a disaster².

The real trick is to find a way to not only make the case that these functions, which I fully agree support engineering, are critical and important in their own regard, but to try to find a way to make the fundamentals of what many of us do day-in and day-out more understandable, relatable, and heck… interesting.

I’m sure no one cares about gmake peculiarities, but “speeding the build up by 30%? Yes plz!”

And everyone loves to bad-mouth capital-P process, but that’s what turns that twenty-step Thing^TM involving four-people stepping on each other editing the same wiki page into a self-serve, two-step process³.

Build/Release, QA, and DevOps Engineers have stories from their every day they could tell you about these sorts of things. All they need is someone to listen⁴.

And to our newest generation of “unsung-heroes”: you’re always welcome at our Unsung Heroes Anonymous meetings.

The coffee’s free.

And the cookies usually aren’t too bad.

_______________
¹ Those claiming “engineers’ paychecks are they ‘thanks’ they get, and they shouldn’t expect any more.” are really missing the point
² Most notably, in my profession, the release engineering team “blocking a release”; long-time QA’ers have felt the brunt of this too, I’m sure
³ If you follow these three simple rules
⁴ And maybe sometimes read a lil’ between the lines…

Despite the article’s tone being a little… Charlie-in-Wonka’s-chocolate-factory-ish, it’s got a lot of interesting tidbits, and is worth the read. It’s especially interesting to see how Facebook’s unique engineering culture has shaped their release engineering process.

(Randomly: I actually worked years ago with Chuck Rossi for a few weeks at VMware before he headed over to Google.)

I did find this bit amusing:

I eventually reached the area where the release engineering team is headquartered. Like the rest of the development personnel, release engineering uses an open space at shared tables. But their space has a unique characteristic: a well-stocked bar.

(Emphasis mine.)

Maybe Google was right all along…

(Lucky!) 0.13, most notably, has the following updates/improvements:

• Entries from your release configuration files can now be coerced by the ConfigSpec class into dictionaries; you can do this be defining an item in the configuration file with the syntax [key1 value1] [key2 value2]
• You can now define command-line overrides for variables; you must set allow_config_overrides to true in your config file to allow this behavior, but after doing so, you can redefine variables from the commandline, like so: quickrelease -p SomeProcess -c myconfig.cfg -DoverrideVariable=overrideValue -DanotherOverride=somethingElse. This is especially useful for processes that are used daily, and thus need minor configuration changes.
• Addition of a chdir() utility method that follows shell semantics; this is a bit of a random feature, but some programs (Perforce’s p4 command line utility, most notably) rely on a correctly-set PWD environment variable instead of getcwd(3) to know which directory they’re operating in; this function provides a compatibility layer for such programs.
• You can now pass either a filename or an input stream to RunShellCommand(), and it will hook this up to the calling process. This is useful for some programs (again, notably, p4) where you can script their behavior by giving them commands via STDIN.

Packages can be grabbed directly from github. (We’ll be publishing “proper” packages for future releases.)

As always, if you run into any problems, don’t hesitate to let us know

And if you’re interested in the project, feel free to follow/fork us on github!

The focus on 0.14 will be something many have been asking for: more documentation, and examples, including a sample set of processes, steps, and configuration to build Firefox!

My professional focus has been on build/release engineering ever since.

At various times, it’s been a difficult road to walk. I truly believe release engineering and configuration management to be an incredibly important part of the software development process, that when done right, can provide an incredible amount of value, from the smallest startups to the largest multinational corporations.

But “RelEng” is often the last part of the process to be designed, and the last part of the team to be filled. It’s often under-resourced, and at that point, it’s usually interrupt driven, (by software that needs shipping). It’s also difficult to change this trajectory, because releng is one of those things that is generally unseen and unheard, until they (or the infrastructure they’re responsible for) become the blocking factor to a ship-date.

How can we approach complex and obstacle-ridden processes safely and successfully every time?

I think there’s a better way to approach these problems.

Friends and colleagues often giggle at me when I make repeated references to aviation in the context of release engineering. Being a pilot, I can’t help but call out the similarities as I see them. But I see so many similarities because I really do feel there is an aspect to that nature of work—whether it be air traffic control, 9-1-1 dispatch, or management of the utility grid—that we, as software engineering practitioners, can learn from and apply to make shipping software more simple, transparent, and predictable.

To that end, today, I’m launching a company: Release Engineering Approaches.

We provide a full range of build/release engineering consulting services including release engineering consulting, build engineering automation & tooling, and DevOps infrastructure support.

Our approach is to integrate those “operational” lessons and practices that so many other industries have (often painfully) learned, to make shipping whatever type of software you need to ship… as simple as possible. Every time.

If consistently and simply shipping your bits are something you struggle with, give us a shout out and let’s discuss how we can help!

To new approaches!

I’m still making my way through it, but I wanted to call out a corollary to one of the facts he covers (which even he calls out as possibly controversial):

An Australian colleague, Steve Jenkin, suggested to me his view of the rate of progress of the software profession. It’s average experience level, he said, has tended to remain constant over time. … [W]hat he meant was: with the explosion of newcomers arriving in this fast-growing profession, the increasing experience level of the growing-olders is more than overcome by the low experience level of the hordes of newbies. As I thought a bit about what Steve said, I came to this thought that I would like to introduce as a corollary:

The wisdom of the software field is not increasing.

If our experience level is not growing, then neither is our wisdom level. In fact, in our mad rush toward the new, we tend to discard much of the old. (For example, software’s newest and most popular methodologies, like Extreme and Agile, tend to make a point of rejecting the accumulated wisdom of the older methodologies.)

This insight particularly spoke to me.

It may be the local environment in which I work, or maybe the (professional) [echo?] chamber I choose to put myself in. Maybe it’s a pattern in the history that repeats every four-to-five years², or maybe it is a fundamental shift in the industry.

In any event, the “DVCS solves all developer problems” and “Cloud solves all infrastructure problems” and “NoSQL solves all scaling problems” and “Rails is the only way to develop a website”-cacophony I often find myself in the middle of… may just be an indication that Glass’ corollary is, indeed, right.

Fred Brooks wrote almost 25 years ago that despite claims to the contrary, no silver bullet exists to reduce complexity of the software development process. And yet it’s become almost fashionable in the space to proclaim as loudly (and proudly) as one can: “Here’s a statement that implies I don’t know the history of the industry in which I practice my craft.”

If true, that’s certainly another point in support of Glass’ corollary.

I’m apparently not the only one who’s noticed the industry may be waking up to what Glass posits was true all along.

What do you think?

_______________
¹ Thanks again for the reco, Ali!
² What I tend to refer to as “an Internet generation”

It’s great to see these conversations happening; I believe this is precisely what should happen in a community when disagreement arises.

One aspect continues to confuse me¹: many of those discussing the issue seem to hold a position predicated on the assumption that Mozilla has claimed support for LGBTQ individuals or (more specifically) same-sex marriage.

I can find no documented substance to this assumption².

Not to call out the pink³ elephant in the room, but in 2008, when the California constitutional amendment involving the issue, Proposition 8, was on the ballot, a number of companies, including Apple, Google, and “numerous biotech companies“⁴ all very publicly stated their official position against the proposition^5,6.

Mozilla Corporation abdicated taking a position⁷.

It’s an open “secret” that Mozilla Foundation board member and Corporation CTO Brendan Eich donated money to support the proposition⁸.

Mozilla, last I knew, does not have any additional protections, stipulations, or benefits⁹ for LGBTQ individuals not already required by California law.

So when I read statements like…

Tim saying:

I’m embarrassed to work for Mozilla right now.

Or Christie saying:

Many members of the community, including myself, as well as members of the general public consider Planet Mozilla a Mozilla news source… . We wouldn’t allow hate-speech there and, we shouldn’t tolerate it on Planet, either. Right now, I feel unsafe and unwelcome at Mozilla.

Or Tim (later) saying:

I feel that my contributions considered less important because I’m queer. If it’s so important to Mozilla to allow speech like Gerv’s speech under the Mozilla banner that it’s worth discarding a percentage of the contributions made by queer and ally employees, then I guess that’s not my decision to make, though I would wonder why and who is accountable for making that trade.”

Or Lukas saying:

It seems like we protect our visual brand identity more than we protect what the Mozilla values appear to be when we refuse to set a minimum code of conduct for participation in our community. Who are we protecting when we do that? Who’s life is enriched by the inclusion of posts that support bigoted points of view?

Or Matej saying:

The point is that it appeared on Planet, which could easily be seen by the general public as an official Mozilla channel that supports the points of view it distributes.

Or Justin saying:

I’d never felt more unwelcome or had my trust broken in a Mozilla setting like that before. I don’t care whether it’s called “hate speech” or “a valid opinion” or whatever, I never want to feel like that again in a Mozilla environment.

Or Al saying:

One thing that I cannot abide is prejudicial actions within that community which go against its basic ethos of inclusiveness and betterment for the good of all.

Or Graydon saying:

Gerv just announced to the internet, using my company’s resources, that my mom isn’t married. And my company is now supporting Gerv’s continued use of our resources (domain name, trademarks, hardware, bandwidth) this way. What shall I tell my mom when I next visit her? “Hi mom, say, did you see that bit where my company endorsed homophobic abuse to deprive you of your marriage?

Or Gregg saying:

That this particular impoliteness carries Mozilla’s domain name and branding is the biggest bummer of all. I want to be proud of my company and my community, and I am not right now.

… I can certainly understand how they all could feel that way.

But given the above history, it’s probably worth examining the assumptions on which those positions are based, and separating out “how Mozilla actually is” from “how we think Mozilla is, and how we would like Mozilla to be.”

If we’re truly concerned about Mozilla’s level of inclusiveness, stance on civil rights, and support for LGBTQ (community) members, then I think it’s clear that a community member’s post on Planet isn’t the first place to be focusing energies.

_______________
¹ Which was hinted at in comments on my previous post
² Someone: PROVE ME WRONG. PLEASE
³ *cough*
⁴ I learned something new!
⁵ Thus, in support of same-sex marriage. Or, at least, not in support of defining marriage into law
⁶ Today, both Microsoft and Amazon publicly support marriage equality in their home states
⁷ As did the Mozilla Foundation; but there may be 501(c)(3) requirements involved there
⁸ Which, to be very clear, is entirely his right, and I am not herein stating a personal opinion on this fact; I am merely reporting it as such, and only in the contexts of inclusion with other Silicon Valley executives’ public statements in the record and one possible explanation why Mozilla may not have followed other technology companies
⁹ e.g. health coverage for domestic partners

If it wasn’t for me, planet.mozilla.org might not be an official Mozilla project module.

That’s a tall claim, so allow me tell a story: see, planet used to be managed by a single person.

It was a thankless job that, apparently, no one else wanted. As was that individual’s purview, content filtering and feed handling decisions were made solely by him. The community wasn’t involved, and there was little-to-no transparency.

This bothered me.

Mozilla was starting other governance initiatives at the time, and this seemed like a perfect example of something to transition to this new system. Asa was the first module owner.

I, being That Guy^TM who opened his big fat mouth, became a peer.

We’ve made what I believe to be the most critically important changes to how Planet is operated: more transparent, with a clear policies to facilitate the community function Planet serves.

And so I’m personally very proud of what raising my voice, via that post in 2007, has achieved: a Planet that, through our work together, has better served the Mozilla community.

Today, some advocated a return to the pre-module days.

They wanted a particular post they personally disagreed with removed.

They wanted feeds changed, so such content would never appear on Planet again.

They wanted a Mozilla community member’s¹ voice quietly silenced.

And as a Planet module peer, that is not something I will advocate for or be a party to.

Ever.

I want to make one thing very perfectly clear: I cannot express the degree to which I disagree with Gerv on this particular issue. But Planet, in its current role, has been a completely unmoderated, open forum for Mozilla community members to express themselves, and I will defend his right to say it, despite vehement disagreement with him.

But let us also be clear: advocating a particular political position in a civil tone is not “hate speech.”

It is not a “safe space” violation².

And the absolute worst thing anyone making such a claim or disagreeing with the position can do is try to censor the discourse.

History, including Planet’s own, has shown time and again: education, empathy, and understanding are not served by the use of “administrative”³ methods to silence an opinion some find unapaltable. And there is an important distinction to be made between defending such opinions and endorsing them.

Fundamentally, today’s events have raised the question about Planet’s role and purpose within the Mozilla community. That’s a fine (and necessary) discussion to have. But it wasn’t a conversation anyone today proposed having.

To anyone in the Mozilla Community who feels “unsafe and unwelcome,” I encourage you to raise your own voice and speak your own position⁴.

I’ll be right here, standing for your right to say it on Planet and in the Mozilla community, too.

_______________
¹ And I would certainly like to hear an argument that Gerv is not a Mozilla community member
² Unless, of course, you consider “safe space” to mean “a space where ideas-I-don’t like are prevented from occurring,” in which case, no: Planet is not a “safe space.” And neither is “The Open Web…”
³ Or harsher
⁴ Mozilla has, in fact, been surprisingly silent on this particular issue