Mental Models: Mattering. (Still.)
Friend and colleague @bear recently wrote up a post1 discussing “magic tricks” developers have started using to store their deploy secrets in git.
Apparently it involves putting data directly into git using “the plumbing,” and encrypting it somehow? The post doesn’t reference specific techniques; if readers have pointers to those, I’d love to read them.
Bear has some thoughts on why this is an idea that will work out poorly for you, and I totally agree with them; you should go take a gander at his thoughts.
I will just add a couple of other thoughts:
-
This is a perfect example of people solving problems that were solely created by a new tool, i.e. git. In other words, securely storing secret data (whether it be deploy keys or software signing keys, or what have you) has been a solved problem with tools like Subversion, Perforce, and even CVS for decades.
A lot of people might come back with “Yah, but those systems suck and git is so much better.” And that’s a conversation we could have, but it’s ancillary to my point, which is: Git has no solution for this problem, and their developers don’t seem interested in solving this class of problem2,3, because it’s not a class of problem the Linux Kernel maintainers have. So they don’t care about solving it in a way that takes into account the body of configuration management engineering knowledge that exists on that particular topic, leaving everyone to reimplement it4 themselves.
I am often harangued as a “git hater” but I will admit: this is one thing that I am unimpressed by git: its (developers’?) answers for so many classes of totally-valid SCM problems is “Don’t do that (because our tool doesn’t support it).”
(It’s worth noting that a lot of the Subversion SCM patterns were based upon what The Subversion Book ™ told you to do; it often recommended you “don’t do that,” but the tool itself still supported the work flow if you did want to do that; I haven’t quite figured out why those who are head over heels for git give it a pass on this particular issue, but will not give any other tool in the history of ever the same “courtesy.”)
-
Bear mentions the following as a reason to not “do magic” with your deploy secrets/app signing assets, but it’s worth reiterating because I think it’s so important: throwing important “secret” data in with the rest of the code, and managing it the same way breaks the mental model those utilizing that data should have about how it’s managed, stored, and used. As bear puts it: “Usage of secret data should have a separate method of retrieval that makes you think about the fact that you are pulling secret data.”
In other words, this is yet another case where the “mental model” matters, in a very critical way.
If you want to scale code, teams, and organizations and do so safely, you must care about the “fuzzy” human factors “stuff”; if you don’t, it will bite you.
It’s not a matter of “if”; it’s a matter of “when.” And it’s interesting the contortions that people who don’t believe human factors matter will go through to ascribe a failure to something else.
The moral of the story: ignore mental models and human factors at your own peril… and “neat-o magic” is probably not the most sustainable way to treat/handle critical business or intellectual property assets.
Hrm, do you have links to how things are solve in, say, Subversion? I was way more dev than ops, and wasn’t paying enough attention I guess. .htaccess?
(My one experience with this with git was a separate private repo and symlinks out of the main repo to the secrets, which were mostly kept in an ini file; this obviously doesn’t work with many situations such as Windows.)
In Subversion, it somewhat depended on the access method (http[s]:// vs. svn:// vs. file://, which I hope nobody used file anymore…), but generally:
http://svnbook.red-bean.com/en/1.6/svn.serverconfig.pathbasedauthz.html
And yes, I’m noticing “Just put it in its own repository” is a Git [anti?] pattern.
(I’ve seen a lot of anecdotal evidence that this scales for the developer who suggests it, and maybe the one or two ops people that have to use it; but much beyond that, becomes a complete mess, usually at a point after the people suggesting it are either off to their next startup or not having to [operationally] support the project anymore.)
Ah, yeah, that was sort of the way I knew for subversion… but it sounded like bear was saying that things shouldn’t be in the same repo at all. Here, I generalized repo to VCS-agnostic… perhaps that wasn’t what he meant? Because for the subversion case, while some directories would be secret, you wouldn’t need to think of it mentally as being apart from the rest all the time.
What I did with the git repo indeed didn’t seem very scalable; luckily (?) for me, that wasn’t the sort of problem we were likely to have – didn’t have the sort of resources to scale. By the point that needed to happen most of the system would need to fixed up
I guess that’s the difference with not working on this professionally…
Hey Mook,
That’s actually a really good point; I’m not sure what bear meant (or was specifically referring to), but I think he was calling out a violation of the mental model of the tool. And with Subversion and Perforce, you can store secrets in the repository without any extra “magic” or changing users’ usage patterns/requirements, and thus violating their mental models.
Yah, I have some anecdotal evidence that the “just create another repo”-[anti]-pattern is the cause of some real scaling issues when using git in a commercial environment.
Most people who claim git does not have this problem are using Github (which, itself, is actually proof of the scaling problem) or they’re at a startup, where they don’t have enough history for them to have run into these problems yet.