By this time, other folks had started to notice the strange gentlemen in the office, asking questions and pointing at things. Sussman waved over two members of his team, Michele D'Netto and LB Denker. Denker explained the testing in production strategy, which they built on top of an A/B split framework.

The Etsy approach boils down to this: When developers want to push a feature, they use an employee mail list to request feedback. Then the programmer pushes the code. It's now on a production server, but through the magic of configuration flags and the A/B split framework, the only people who will execute that code will be logged-in employees.

After reasonable testing and feedback, the programmer can promote the code to a larger group, or to a random portion of users, which eventually inches up to 100 percent.

"It's not just the GUI," Rembetsy said. "We often add in features that are 'dark,' or behind the scenes. Right now, for example, we are adding a second logger. We can add storage, logs, even swap databases, and actually go ahead and duplicate reads and writes to the second DB, measuring performance. Forget about performance worries -- by the time we cut-over, we've seen the thing work in production for a month."

I asked Sussman, if they have this great framework, why the graphs? I mean, shouldn't they know things will work? "Well, sure, in theory," he said. "And even, most of the time, in practice, too. But humans make mistakes. It is always possible to think you put a feature in dark, miss a config flag, and , it goes out to the world. So you watch the graphs, you figure out the problem, you fix it, and you drive on.