Building Central: Pair Programming

In the summer of 2013, CoverMyMeds had agreed to write a workflow system called Central. Due to a few miscalculations, we ended up with far less time than we’d wanted, and found ourselves in the classic “too much to do / not enough time to do it” dilemma.

One of the techniques we used to help us get through this was pair programming, a classic software building technique where two developers work on the same thing at the same time. If this whole concept is new, Wikipedia has a typically bland description, c2 has a more irreverent but potentially more helpful analysis, and there’s tons of books that can help you get off the ground.

Here’s how we did it during Central’s formative first few months: my pair, B, and I would sit at a table. B would be in front of a computer, and I would be in front of a keyboard/mouse/display connected to that same computer and mirroring its display. That computer had Central’s code checked out and working development environment set up. It also had an editor set up in a way that both B and I could use it.

Typically we’d ping-pong — one of us writes a test that fails. The other person then writes the implementation that makes the test pass, then writes the next failing test and passes control back. While one person is busy typing, the other is reviewing the code live, making suggestions for improvements or asking questions.

That’s a fine technique, but it doesn’t always hold up. Sometimes, if one person had a solid vision in their head for where to go next and the other didn’t, we’d just let the one person keep control for a while. This isn’t a good way to work long-term, because one person ends up a passive observer eventually, but it works well when the pair is trying to get unstuck or one person knows a language/module better than the other. I learned a ton by watching B work in libraries I hadn’t had a chance to use yet.

Also, when one person is having a shit day and isn’t able to keep themselves focused, the other can gently nudge them into participation. Maybe I stayed up too late and now I’m cranky and tired. Having your primary display be someone else who’s working on the thing you’re both responsible for means it’s hard to spend 40 minutes reading ancient jwz blog entries, especially when someone’s just written a new test and is asking you how you want to write the implementation.

We both had our personal laptops off to the side, so we could monitor incoming email/IMs, or handle personal business. You can pair on one person’s personal machine if you connect up a second keyboard/mouse/display to it, but that always makes me nervous. My development environment probably won’t be immediately useful to you, and I’m constantly anxious about an embarrassing personal message notification popping up. Having a neutral third machine was incredibly helpful, but it did require that we shared an editor or IDE. We got lucky, since we both used a similar vim setup we could just pick one and run with it, but that can easily be a huge stumbling block if you’re not that sympatico.

We did have a strict pairing policy, if we were both free and in the office we’d be pairing. This helped us keep focus. If one of us was out at a meeting or something, the other could take solo duty and keep forward progress going. Sometimes I’d take the opportunity to take my personal laptop into a common area, chill myself out, get some of our stuff done, and push it up to a branch we could later pull down on our pairing machine and review together.

You can do this kind of pairing remote, but as with all things remote, it’s not quite the same. Input lag for the remote party can range from “annoying” to “this is impossible screw this,” and even the best video conference isn’t a replacement for being right in the room with someone. That said, it’s not impossible. If you can’t find a neutral third machine to screenshare into, alternate who hosts the screenshare each day, and be strict about communication — my experience with remote pairing is that it’s too easy for person sharing their screen to just keep control for most of the session, with the occasional “hey this look good to you?” That’s no way to pair.

When we first suggested to our manager that we ought to pair on Central, he was skeptical. He expected two developers working in parallel would double our throughput, even though I know he’s read The Mythical Man-Month. And he wasn’t necessarily wrong, two developers working independently will produce more software than if those two pair, no doubt. I thoroughly believe that we’d have ended up with an awful train wreck if we’d both independently went off and tried to build out this new system. We had to learn a lot about our customer’s workflow and business requirements, and having us both on the same page was invaluable.

Finally, some folks are insistent that entire departments should pair, and that the pairs should be rotated every n days, usually at the end of a sprint or something. That can do a lot for team development and shared understanding of different codebases, but we didn’t do any of that. We needed to build and ship this software quickly, and if we’d been forced to rotate, we’d be putting team dynamics above providing value to our customers. At this stage in CoverMyMeds’ growth it wouldn’t have made any sense.

So yeah, don’t be afraid to try pairing.

On Picking Your Tech Stack

Part of a comment from Hacker News, which I should probably stop reading:

The biggest issue with ORM [object-relation mappers] isn’t the solution they came up with, half baked and buggy as they are, but the problem they are actually employeed [sic] to “solve”: developers afraid of SQL. Whether they can’t write it, can’t learn it, or just straight hate the string representation in their code, it’s the main reason people sign on for ORM.

In the early CoverMyMeds days, one of our most senior developers was not shy about telling you what they thought of ORMs, or the weak developers who relied on them. He was convinced that using a tool to write SQL queries for him took his control away, and that knowing precisely what queries powered each page was the best way to ensure performance and maintainability.

Thank goodness that person wasn’t able to make tooling decisions for the entire company, or we’d have been hosed.

Actually, the way we made technology decisions at CMM was pretty decent, and completely informal. Each project was allowed to pick the tooling that they wanted, given a few truths:

  1. Our infrastructure/production operations folks could support it. Ruby/Rails had become a safe choice, but folks still talk about how one dev wrote a service requiring MongoDB, which we’d never used. A week before launch they had to frantically re-architect the whole thing; ops had refused to try to support MongoDB with no lead time.
  2. You weren’t the only developer that knew it. We’d had an issue where some Python apps were launched to production, but only one person knew Python. That became problematic quickly when that person didn’t feel like they could take a day off.

Again: these were informal, the way everything was back then. There wasn’t a policy/procedure document that all the developers had to sign or anything, just an understanding amongst everyone. We knew we were all ultimately responsible for the success of our projects. This meant that my ORM-hating colleague was free to choose to hardcode every SQL statement his app wrote into a module somewhere while the rest of us used ActiveRecord.

Which, for the record, won out. Every developer at CoverMyMeds was required to be conversational in SQL, but not needing to write every instance of SELECT TOP 1 column FROM table ORDER BY sort_order ASC allowed us to focus more on what value we were supposed to be providing.

In an early-stage company, the technical decisions matter far less than a 25-year-old software developer wishes they did. You, your team, and your company should all be focused on how to solve a customer problem today. If your technology decisions are making you do more work but it’s not in service of solving a customer problem faster, they’re probably the wrong choices.

How We Handled Our Toxic Friend

If you work in tech, you know at least one person who you will never work with again

Z is one of the best software developers I’ve ever met. They know more than I’ve forgotten about data structures and algorithms, and they can ship features and deliver value without pause. They were a force in early CoverMyMeds, setting the tone and culture of our early development team. It’s safe to say that CMM wouldn’t be where it is today without their efforts.

That said, Z was kind of abrasive. They were always the type of person who held strong opinions — many software developers are, comes with the landscape — but over time it started to get worse. At first, it was an off-color joke here or a code review that was friendly but unyielding. Then it was a code review that was unyielding but now felt kind of like a harsh attack.

Eventually, it got bad. Maybe it was bad already and we were all just giving Z leniency because they were our friend and we were in a make-or-break phase of the startup together. But it became too much. Shouting in meetings at whoever was opposed to their ideas, up to and including Matt, the CEO, who was our friend. Actively berating other developers for writing what they deemed sub-par code. Being awful to anyone not in tech who was trying to work with them, such as our account managers. Z was incredibly valuable to the company as a developer and had tons of institutional knowledge baked into their head (not to mention their user id hardcoded in places in the system for debug/admin purposes), but we were all at our wits’ end.

We had the classic “toxic team member” problem on our hands. What did CoverMyMeds do?

First, we — and when I say “we” here I don’t mean me personally, I wasn’t management and therefore not a part of these decisions — remembered that Z was our friend and a human person. When you employ (or even manage) someone, you hold their livelihood and career future in your hands, a responsibility that you cannot take lightly.

Z’s manager made sure that they understood clearly that their behavior was unacceptable, and that if it didn’t improve there’d have to be action taken. Nobody at the company was irreplaceable. That’s always the first step.

Then we lightened their workload. Z was the type of person to work 18-hour days without being asked, and it helped ship a ton of stuff on time, but it had taken its toll. We made sure they weren’t put under harsh deadlines. We ensured their workload didn’t require them to work after-hours — although they continued to anyway because they didn’t know how not to.

Finally, we pulled out the big guns. We forced a multi-week vacation on them, ordering them not to VPN in to get anything done and to just chill themselves out. We offered to pay for therapy, help them work through some of their stuff. (Side note: probably everyone could stand to talk to a therapist every now and then.)

Nothing worked, and on what I remember being a grey and dreary day, after another bout of shouting at friends in a meeting on subjects that didn’t warrant that kind of passion, Z was fired.

We tried everything we could imagine, but Z’s original conversation with his manager was true. Matt reminded all of us of that at the all-hands that was called right after. Matt was torn up, clearly unhappy that things had come to this with someone we all liked, but he was clear: we’ll try our best with everyone, but nobody is irreplaceable.

So recently, when the CEO of a company I’m invested in and advise told me that they’d had to fire their lead developer that morning because of awful behavior, I asked what they tried prior to letting them go. The CEO said they’d talked, they’d tried to explain, there were meetings, but the company was small and nobody could stand it any longer.

And I told them they’d done the right thing.