So, in the last thing I posted, I wrote about how fixing problems on live web sites feels like fixing space robots. At Mozilla, though, things have been getting steadily better and more interesting:

There’s still the sysadmin-as-remote-pair-of-hands-driven-by-trouble-tickets thing going on, but less so every day. More and more, we’re getting into continuous deployment. Our sysadmins are really smart cookies: They’re steadily replacing themselves with scripts of varying sizes. That way, they can get on to more interesting things. In fact, just recently, they gave us a great Big Red Button that deploys code from a git repository to the live site on demand. And, for some reason, I’m one of the people who got a key to it.

Okay, maybe not that exact button, but ours is very keen. Of course, when it broke once, I felt like the Pakled on the right (and not Geordi):

But, the Big Red Button hasn’t broken very often. So, I’d rather be a Pakled some of the time than pretend to be a rocket scientist all of the time.

Oh yeah, and we’re using GitHub a lot these days. It’s crazypants awesome for collaboration, and it fits right in with our style because we want to do everything in the open anyway. We’ve even somehow lured people who don’t normally code into submitting Pull Requests. And, it’s amazingly powerful to refer to specific lines of code and commits by URL in IRC or email.

I’ve also been getting a lot better at building development environments, using Puppet and Vagrant and Amazon EC2. I haven’t quite gotten to the point where I can borrow IT’s Puppet manifests directly – ie. those scripts with which sysadmins are replacing themselves.

But, we are maintaining the recipes for spinning up dev servers right alongside the code. Our READMEs are shrinking, rather than growing – and yet we’re still adding more services and dependencies. I no longer really fear totally jack-knifing my dev rig, because I can just toss it and conjure up another in the time it takes me to run to lunch.

Let’s see, what else? I’m sure I’ll think of something. Until then, SHIP IT. (The post, I mean.)