Continuous Integration

Release Schedule

May 2020
S M T W T F S
« Nov    
 12
3456789
10111213141516
17181920212223
24252627282930
31  

Branches

Twitter

© 2020 The Sober Build Engineer | Simply ship. Every time.

Simply ship. Every time.

The Data-driven Dilemma

01/14/2014

Data-driven decision making. Since appearing on the tech scene about a decade ago1, it’s seemingly become all the rage in Valley management culture.

It’s a pillar of the “lean startup” ethos. It’s alliterative. And it’s arguably become the dominant decision making paradigm, trumping even time-honored, classical engineering analysis, for established tech companies and startups alike.

It’s an alluring proposition: it purports to remove ego from discussions, making the data you bring to the table (and can explain with analysis!) the first class citizen, not your title. It tries to open up the paths of communication, so the flow of decision making isn’t top-down, allowing ideas to flow both ways. Lots of research today, especially in the lean manufacturing space, indicates that the people on “the factory floor” are make decisions that result in better outcomes than managers reading reports in another building and dictating policy and process from there.

After years of employing the strategy, some have been able to “adapt” the method to re-establish the patterns data-driven decision making was supposed to address. I’ve seen this firsthand, but what I found interesting is these (anti-)patterns had been observed by others too.

Some interesting examples of the contortion of the data-driven mantra:

Read More

__________    
  1. From what I can tell, anyway; if I didn’t know better, I’d say it seemed like Stanford had a data-driven decision making course it required all its MBA and engineering management masters students to take↵
betas._V371004456_

Skip the Betas

01/10/2014

When I saw an ad1 for Betas, Amazon’s original comedy series about a group of Silicon Valley millenials building the next disruptive dating app, my curiosity was piqued.

But after watching the first two episodes2, I have to say: Betas is completely unwatchable.

The first two episodes cover what you’d generally expect: character exposition, contrived situations serving to bring our protagonists together and establish our antagonists, the awkward setting of social/sexual tension and, of course, setting the story arc for the season: the trials and tribulations of building of a mobile application in a startup incubator.

Unfortunately, the execution is all too contrived: the characters, from the shy, quiet Indian wunder-engineer to the pretty, upstart early 20-something “CEO” who has just a touch of “Aspies” to the jamming-with-Moby venture capitalist aren’t particularly innovative. The “comedic” situations our stereotypes find themselves in aren’t inspired—posing as Larry Paige to crash a VC’s house party? Really?—nor is the character interplay or “tension”: at times, I feel like I’m watching an episode of Glee, except without the singing.

Or the diversity.

If, like me, you were hoping a show like this might tackle that particular issue our industry finds itself grappling with3, you’d be correct, though probably not from the angle you might have hoped.

Betas ticks all of the boxes off that checklist:

Sexism:

We’ve got free broad band, free snacks, and not to mention: those bangin’ ass hotties with the bargain hunting app. [Camera pans to a group of five women around a table; only one of them has a laptop.]

(Because obviously, women entrepreneurs’ most noteworthy trait is their “ass-banging’ness.”)

Oh, and the antagonist-team’s rival app? “Valet-me”: from the makers of “Tit-stare.”

Racism, check:

I’ve got a Bitcoin on Bollywood.

Uttered as the character raises a physical coin (Haha, get it? Currency!), referring to her Indian—red dot, not feather, a riff we hear in the same episode—teammate.

Drinking culture? Our VC has you covered there; his new incubator class is informed that at the kickoff party:

Remember: Intoxication is mandatory.

Everyone claps, of course.

Oh, and let us not forget ageism:

You know I’m 35; it’s like 95 in Valley Years…

Now I know this is a sitcom, and the point is to get a laugh. A lot of things were said that… were sort of questionable, but I didn’t bother noting them because other sitcoms have made equally tasteless jokes for an (equally) cheap laugh.

The most surprising thing about watching Betas is that I realized I was actually becoming frustrated and angry while watching it. It took me awhile to figure out why: Betas’ characters do a really good job of re-creating archetypes of our industry, so much so that they remind me of people I’ve met. But the problem is… none of those people are likable.

Co-creator Josh Stoddard said: “We were looking at this culture, and it is pretty easy to parody.” But this show really isn’t a parody, at least not in the sense IT Crowd or Hackers were. Those had characters that maybe frustrated us, but they were all likable in some way or another because they possessed certain aspects of ourselves that we could identify with.

I don’t identify with any of these characters. Nor do I want to. Heck, I wouldn’t even want to have coffee with them.

The other frustrating part is I couldn’t really figure out who this show was supposed to be for; I came up with two possibilities:

Read More

__________    
  1. In print; on a Muni bus shelter no less!↵
  2. A small sample size, yes, but Amazon offers the first three episodes free, and then you have to pay; so they have to hook you in three↵
  3. What group has Paul Graham decreed incapable of programming today?↵

Old Checksums Collide Hard

01/08/2014

Our industry still insists on using MD5 checksums. And I don’t know why.

Since 2004, it’s been widely reported that “the security of the MD5 hash function is severely compromised.” When talking about collision attacks in hash functions, the answer to whether or not they’re broken—and thus useless for the sole purpose we employ them—is not unlike answering whether or not one is pregnant.

And yet, I still see MD5 checksums floating around all over the place, providing a false sense of security.

It’s interesting to speculate on why this might be.

Obviously, there’s the inertia argument: release infrastructure is one of the least exciting places for the business to invest in, so if you’re publishing checksums at all1, changing the hash functions—”What do those even do? And does any customer use them?”—probably isn’t the top of management’s list.

(It doesn’t help that OS vendors don’t make this easier either: OS X Mavericks still ships an command line md5 tool, but nothing to easily calculate SHA12. And you wanted to easily checksum a file on Windows? Good luck with that.)

Perhaps it has to do with the fact that (oddly?) most people just don’t seem to care to validate that the bits running on their machines are what the software company intended to ship? I guess they assume that nobody is tampering with their Internet traffic? It’s obviously a point of consternation when their machine goes rogue on them, but… that never actually happens, right?

Anyway, ever since 2006, I’ve been suggesting to my clients that they use SHA1 checksums, and forgo even publishing MD5. But that advice may not be correct anymore.

At the recent Chaos Computer Club conference in Germany, researcher Rüdiger Weis presented evidence (English translation) that SHA1 should be taken as effectively broken too.

The news is particularly interesting, especially in light of the recent revelations regarding the NSA: part of his justification for arguing its compromised status is that SHA1 is in the same family as SHA03, which was developed by the NSA, without a published technical specification; it, too, is considered broken. He also notes the modification to SHA1 was made by the NSA without any technical explanation.

(Weis puts it more colorfully: “I would personally rather put a wager on Britney Spears’ virginity than as to the security of SHA-1.”4)

His recommendation for a hash function? SHA35. Unfortunately, it’s less clear what we should do.

Security guru Bruce Schneier says SHA3 isn’t necessary, and SHA512, a member of the SHA2 family, is as of 2012 “still looking good6.” It doesn’t help that even if we wanted to use SHA3, I couldn’t find a utility on Linux implementing it; sha512sum is implemented in coreutils 8.x.

Maybe we’ll all end up moving toward what Gentoo does, and brute-forcing it by validating multiple checksums, from multiple families; portage currently checks: sha256, sha512, Whirlpool and file size.

Either way, anyone care to wager on how long it’ll take for us to stop relying on SHA1 checksums to validate bits? (That is… if we ever do…)

__________    
  1. Good on you!↵
  2. Yes, I know you can use openssl for that purpose, but it’s still annoying↵
  3. And SHA2, for that matter↵
  4. Translation mine, with the help of Google Translate↵
  5. For encryption, he recommended against ECC, in lieu of 256 bit AES, 4096 RSA DHE and use a 512 bit hash function↵
  6. I guess he’s less worried about NSA-backdooring than Weis↵
OS/2 Warped

A WARP’d View of History

12/15/2013

Ars is carrying an interesting historical article on the history of IBM’s OS/2. The article describes it somewhat provocatively (though, probably equally realistically) as “half an operating system.” Definitely worth the read if you’re into classic operating systems or computing history.

I knew most of the information, largely from Robert Cringely’s PBS series Triumph of the Nerds1, but one thing I didn’t know about IBM’s repeated marketing foibles with the product:

OS/2 version 3.0 would also come with a new name, and unlike codenames in the past, IBM decided to put it right on the box. It was to be called OS/2 Warp. Warp stood for “warp speed,” and this was meant to evoke power and velocity. Unfortunately, IBM’s famous lawyers were asleep on the job and forgot to run this by Paramount, owners of the Star Trek license. It turns out that IBM would need permission to simulate even a generic “jump to warp speed” on advertising for a consumer product, and Paramount wouldn’t give it. IBM was in a quandary. The name was already public, and the company couldn’t use Warp in any sense related to spaceships. IBM had to settle for the more classic meaning of Warp—something bent or twisted. This, needless to say, isn’t exactly the impression you want to give for a new product. At the launch of OS/2 Warp in 1994, Patrick Stewart was supposed to be the master of ceremonies, but he backed down and IBM was forced to settle for Voyager captain Kate Mulgrew.2

I guess this explains OS/2 Warp’s funky logo.

(The article also includes an interesting ad wherein an aging (and “somewhat befuddled”) cast of M*A*S*H is inexplicably surrounded by a bunch of IBM PC boxes.)

Apparently Reimer does a lot of articles on historical OS’s; you should check out his Ars publication list if that’s your cup of tea.

__________    
  1. Which author Jeremy Reimer specifically notes that he took much of the information from and “[view] multiple times during research.”↵
  2. I don’t know why they’re giving panning Captain Janeway; she’s a great jailed Russian mobster↵
chef

A Recipe You’ll Never Forget

09/24/2013

Last week, I attended Opscode’s public Chef training held here in San Francisco.

As a release engineer, most of my career has been spent wrestling with configuration management. And while I’d dabbled a bit with Chef, I hadn’t really “put on the apron” to graduate from sous-chef yet.

So I decided it was time to get out the cookbooks and do that.

In the recent Ship Show episode on training, we debated whether or not training is valuable at all. Sascha asserted that you can often obtain the same benefit by picking a real-world problem to solve, holing yourself up in a room with a book on the subject (or the Internet), and just working on that it until you’ve solved it.

I think that can be a way to do training, but I remain unconvinced that it’s particularly efficient, especially for a team. And, in the absence of a domain expert, I think it’s quite possible to miss some of the valuable ah-ha moments that illustrate concepts that are really necessary to be successful with the tool. (Or avoid hating it, because you wasted two hours on some slightly-misunderstood concept.)
Read More

The Subversive Repository

08/05/2013

This month’s Vanity Fair has a story on an odd criminal case every programmer should familiarize themselves with, if only for their own safety.

The story involves Sergey Aleynikov’s (unfair, some might say) prosecution for allegedly stealing high frequency trading source code from Goldman Sachs. From the article (and emphasis mine):

In April 2009, Serge had accepted a [new] job …, but had remained at Goldman for the next six weeks, until June 5, during which time he sent himself, through a so-called “subversion repository,” 32 megabytes of source code from Goldman’s high-frequency stock-trading system. The Web site Serge had used (which has the word “subversion” in its name) as well as the location of its server (Germany) [the FBI] clearly found highly suspicious.

… “I thought it was like, crazy, really,” [Serge] says. “[The FBI agent] was stringing these computer terms together in ways that made no sense. He didn’t seem to know anything about high-frequency trading or source code.” For instance, Serge had no idea where the “subversion repository” was physically located. It was just a place on the Internet used by developers to store the code they were working on. “The whole point of the Internet is to abstract the physical location of the server from its logical address.”

I find it interesting that reporter Michael Lewis fails to make it extremely clear to readers that, in this case, failing to use the proper noun form of “subversion” is disingenuous.

(If the picture Lewis paints of the FBI’s conduct in investigating this case is to be believed, their misuse is not surprising; but he misses out on an opportunity to contextualize this misconduct by only hints to readers that the word refers to a software program, not some “repository of iniquity.”)

In any event, there are a lot of reasons to use Git(hub), but “not confusing FBI agents on a crusade to put you in jail” is certainly a new (and compelling!) one.

Not X. But Not Quite Y.

07/26/2013

It’s pretty much a given that generational demography turns out to be a softer science.

I am most commonly thrown in with “The Millennials” (which is what they apparently settled on for the initially-named “Generation Y”), though anyone who’s talked with me on the subject knows I don’t really feel connected to that generation, its ideals, or its accepted behavioral norms.

I had always attributed this to where I’d grown up1, but last week, a couple of friends pointed me to a couple of articles illustrating that I’m not alone in this unwillingness to fully embrace Millennials, despite not feeling like a Gen X-er, either.

Dorre Shafrir’s “Generation Catalano” second paragraph nailed it for me:

The Carter babies—anyone born between his inauguration in January 1977 and Reagan’s in January 1981—are now 30 to 34, and, like Carter himself, the weirdly brilliant yet deeply weird born-again Christian peanut farmer, this micro-generation is hard to pin down. We identify with some of Gen X’s cynicism and suspicion of authority—watching Pee-Wee Herman proclaim, “I’m a loner, Dottie. A rebel,” will do that to a kid—but we were too young to claim Singles and Reality Bites and Slacker as our own (though that didn’t stop me from buying the soundtracks). And, while the proud alienation of the Gen X worldview doesn’t totally sit right, we certainly don’t yearn for the Organization Man-like conformity that the Millennials seem to crave.

I’m not sure I like the name she proposes—what’s so wrong with reclaiming Generation Y?—but I certainly identify with sentiment, and the article accurately captures what a lot of us (surprisingly!) seem to feel.

Switching gears to a more whimsical treatment of the topic, I got linked to “22 Signs You’re Stuck Between Gen X and Millennials.

Signs 22, 53, 64, 75, and 136 are certainly true; 37, 108, and 219 feel right. And 110, 1111, and 1912 are all factually accurate.

It’s nice to see that I’m not the only one who feels a bit too young to be an X-er, but “generationally challenged” when it comes to identifying and interacting with “Millennials.”

All of this discussion on “micro-generations” reminded me of another discussion I had with a friend that sliced and diced this issue in terms of technology: his posit was there exists is a [micro-]generation, of which we’re both a part of, of people who are old enough to have learned and dealt with an specific era of technology, but are young enough to have also learned the (beginnings) of the new technologies.

My favorite example of this concept is the card catalog: I can remember having to learn to use it in elementary school, but no one ever did after 5th grade. Records are another: I grew up with them13, but like most teen-aged Millennials, I started collecting compact discs when my age rose into the double-digits.

I remember an Internet where gopher, email, and FTP ruled the tubes, and that sweet sound of a handshake produced a very real emotional response, even though when I graduated high school, “the Internet” was pretty much solidified in the cultural consciousness as only “w-w-w-dot-whatever-dot-com.”

In other words, the emerging, world-changing technology isn’t foreign to us, as it was been to X-ers and Boomers, and we were able to incorporate it into our lives and identities, but we still have vivid memories of “the old way,” which I think imbues a sense of both perspective and respect not only for the way things were, but that such a world could exist without ubiquitous cell phones and fast, always-on Internet, bringing us music no one buys physical media for anymore.

Perhaps it is this perspective that contributes to the feeling of being between Gen X and the Millennials, and yet not of either of them.

__________    
  1. And, to a lesser—but still notable extent—how I’d been raised↵
  2. You don’t identify as being either Gen X or Gen Y, but when pressed, you’d rather be seen as a Gen Xer↵
  3. You can’t name more than three ’90s Nickeleodeon shows.↵
  4. You have no idea who Lizzie McGuire is↵
  5. Gen X had The Electric Company, Gen Y had All That, but you had You Can’t Do That on Television.↵
  6. And you didn’t get a cell phone until you were at least 22… Meanwhile, your little sister had already had one for a couple years↵
  7. You feel your identity is slightly mixed up, like millennials are the bubbly, naive side of you, and Gen X is the cynical, pessimistic side.↵
  8. Whenever Time has done one of these huge generation pieces, you never feel as though they’re talking about you.↵
  9. You feel torn between valuing the old and completely embracing the new↵
  10. You were born between 1975 and 1982.↵
  11. When you hear the word “turtle,” you think of this↵
  12. [You actually drank] Zima↵
  13. And a skipping record always produced uncontrollable giggling from me as a toddler↵

Mental Models: Mattering. (Still.)

07/23/2013

Friend and colleague @bear recently wrote up a post1 discussing “magic tricks” developers have started using to store their deploy secrets in git.

Apparently it involves putting data directly into git using “the plumbing,” and encrypting it somehow? The post doesn’t reference specific techniques; if readers have pointers to those, I’d love to read them.

Bear has some thoughts on why this is an idea that will work out poorly for you, and I totally agree with them; you should go take a gander at his thoughts.

I will just add a couple of other thoughts:

  • This is a perfect example of people solving problems that were solely created by a new tool, i.e. git. In other words, securely storing secret data (whether it be deploy keys or software signing keys, or what have you) has been a solved problem with tools like Subversion, Perforce, and even CVS for decades.

    A lot of people might come back with “Yah, but those systems suck and git is so much better.” And that’s a conversation we could have, but it’s ancillary to my point, which is: Git has no solution for this problem, and their developers don’t seem interested in solving this class of problem2,3, because it’s not a class of problem the Linux Kernel maintainers have. So they don’t care about solving it in a way that takes into account the body of configuration management engineering knowledge that exists on that particular topic, leaving everyone to reimplement it4 themselves.

    I am often harangued as a “git hater” but I will admit: this is one thing that I am unimpressed by git: its (developers’?) answers for so many classes of totally-valid SCM problems is “Don’t do that (because our tool doesn’t support it).”

    (It’s worth noting that a lot of the Subversion SCM patterns were based upon what The Subversion Book ™ told you to do; it often recommended you “don’t do that,” but the tool itself still supported the work flow if you did want to do that; I haven’t quite figured out why those who are head over heels for git give it a pass on this particular issue, but will not give any other tool in the history of ever the same “courtesy.”)

  • Bear mentions the following as a reason to not “do magic” with your deploy secrets/app signing assets, but it’s worth reiterating because I think it’s so important: throwing important “secret” data in with the rest of the code, and managing it the same way breaks the mental model those utilizing that data should have about how it’s managed, stored, and used. As bear puts it: “Usage of secret data should have a separate method of retrieval that makes you think about the fact that you are pulling secret data.”

    In other words, this is yet another case where the “mental model” matters, in a very critical way.

    If you want to scale code, teams, and organizations and do so safely, you must care about the “fuzzy” human factors “stuff”; if you don’t, it will bite you.

    It’s not a matter of “if”; it’s a matter of “when.” And it’s interesting the contortions that people who don’t believe human factors matter will go through to ascribe a failure to something else.

The moral of the story: ignore mental models and human factors at your own peril… and “neat-o magic” is probably not the most sustainable way to treat/handle critical business or intellectual property assets.

__________    
  1. Which, I hate to admit, I’d had sitting in a tab for a couple of weeks, before having had time to read it!↵
  2. Assuming you buy that it’s even solvable at all↵
  3. Git would really have to solve the (centralized) identity problem first, I think…↵
  4. Usually poorly↵
Newer Posts
Older Posts