Data-driven decision making. Since appearing on the tech scene about a decade ago¹, it’s seemingly become all the rage in Valley management culture.

It’s a pillar of the “lean startup” ethos. It’s alliterative. And it’s arguably become the dominant decision making paradigm, trumping even time-honored, classical engineering analysis, for established tech companies and startups alike.

It’s an alluring proposition: it purports to remove ego from discussions, making the data you bring to the table (and can explain with analysis!) the first class citizen, not your title. It tries to open up the paths of communication, so the flow of decision making isn’t top-down, allowing ideas to flow both ways. Lots of research today, especially in the lean manufacturing space, indicates that the people on “the factory floor” are make decisions that result in better outcomes than managers reading reports in another building and dictating policy and process from there.

After years of employing the strategy, some have been able to “adapt” the method to re-establish the patterns data-driven decision making was supposed to address. I’ve seen this firsthand, but what I found interesting is these (anti-)patterns had been observed by others too.

Some interesting examples of the contortion of the data-driven mantra:

• The Veiled Ego: a colleague relayed a story about his org’s engineering leader, who beat the “data-driven” drum endlessly, in meeting after meeting. The engineering decisions made mirrored what the leader suggested first, before the data was in, but the data usually seemed to support his conclusions, so my colleague never thought twice about it. The guy was genuinely smart, after all. It never occurred to him that there was more going on until he had a meeting about a strategic decision. It had to do with a large investment in either virtualization or hardware for a build farm-related task. The VP argued for physical hardware; when faced with doubt from my colleague and his team and a reminder that “we’re data-driven here, right?” he relented. “Ok, go get the data.”

  A week later, they returned, data in hand. It unequivocally pointed to virtualization: it would be faster and somewhat cheaper, plus vastly easier to manage. At this point, you’d imagine the VP would change his position, right? To my colleague’s surprise, he threw a tantrum.

  A raised-voice, pacing-around-the-room, and finally leaving-the-meeting-early tantrum. Without any (formal) decision made².

  This experience caused my colleague to understand that the leader had been calling the decisions as he saw fit all along. Since he was a smart guy, the data supported him around 80-90% of the time. When an underling had an idea that contradicted the his already-made decision, a demand for data was called. But when the data disagreed with his opinion, which was seldom enough to go undetected when viewed on short time-scales, the VP not only balked, his ego got involved and he did it in a very obvious way.

• Data-driven paralysis: another colleague shared an interesting dynamic he’d run into with various organizations subscribing to data-driven decision-making, a riff on the previous anti-pattern: whenever anyone had a new idea they wanted to explore, a line everyone had learned to use to shut the idea down? “Well, we can’t do that: we have no data on it.”

  Obviously, for new ideas, there will seldom be actionable data, and some research or exploration needs to be done. But using the lack-of-data—because all decisions must be data-driven—to shut down ideas is certainly a distortion of the principle. And yet, in these organizations, it was a strategy that worked well for those wanting to dismiss new ideas or politically undermine the success of an idea or plan gaining traction, but which they personally disagreed with.

• The last one is probably the most visible anti-pattern, and one I like to call Data-mining the Mona Lisa: certain human endeavors—art, typography, graphic design, music, photography, and the like—have a component of them that is objective, and can be analyzed. We know certain chords and color schemes are “more pleasing” to humans than others. But beyond that, those works, which we incorporate into our software products, are largely based on aesthetics. Individual artists will have differing aesthetics.

  When you reduce an artist’s work to A-B testing (or sometimes, A-AB testing), you’re basically telling your creatives³ you don’t value their opinions, and you could have hired a computer to paint-by-number, for all you care.

  Obviously, because aesthetics do differ, it’s important to hire someone whose aesthetics you[r organization] generally finds palatable, but once you’ve done that, you need to trust them. And there’s no faster way to annoy an artist than to tell them that their aesthetic value judgments, honed by years of practice of their craft, are worthless, and therefore you’ll be A-B testing them.

  The most visible example of this was Douglas Bowman leaving Google.

There are likely myriad examples of the ways in which leaders say “We’re 100% data-driven,” but employ a “But really, just do what I say and don’t ask questions” methodology in practice⁴.

So next time you hear “We’re a data-driven decision making organization,” don’t take it at face value. Stand back, watch the mechanics of how a few decisions get made, especially contentious ones.

This will reveal whether it’s a slogan—not unlike “We’re a DevOps organization” or “we only use an agile-based methodology for managing our software projects”—that everybody has learned to they need to repeat to operate within the environment, or whether it’s a practice the organization really buys into, and practices to make the best decisions, whether the idea is from the CTO or an intern.

__________    
1. From what I can tell, anyway; if I didn’t know better, I’d say it seemed like Stanford had a data-driven decision making course it required all its MBA and engineering management masters students to take↵
2. Actually, the VP said “Well, do whatever you want,” but my colleague had learned the translation on that was “Just shuttup and do what I say.” (Or as the VP often put it “Disagree, but commit.”)↵
3. Note that, like Mike Monteiro, I don’t like that term, because it’s been used to devalue their work↵
4. And I’d love to hear them; please feel free to comment!↵