How to Make the Most of Good Metrics

Do you know which metrics make sense for your product development efforts?

Management loves metrics, and there are lots of metrics offered up in the ether of “developer best practices” for software development. Whether they’re considered a necessary evil or a useful tool might depend on the company you talk to.

There are numerous varieties of metrics and each one is usually intended to help gauge the performance of a particular process. For software companies, metrics can help them measure their processes, projects, and products. But if they’re used improperly, they can also do more harm than good.

I recently exchanged e-mails on this topic with a CTO of one of the expansion stage software companies in our investment portfolio.

We discussed a particular software development metric that I think can be helpful for some teams: number of builds run per day. The CTO wasn’t sure he saw the point in that metric, but I responded with a few points:

• If a team is doing continuous integration to best practice, the developers should – in most situations – be doing multiple commits per day as they complete tasks and stories. So, if the metric is very low, it may point to an opportunity for improvement in identifying defects or other issues earlier in the spring.
• Trying to measure the metric may point to the team not doing continuous integration. Asking for it may spur discussion around the metric and, again, lead to significant improvement.

My argument was directed toward a hypothetical team with very specific needs or issues that could be measured. The truth is that the metric we discussed may be completely useless for teams that have already been doing best practices in continuous integration. After all, that metric is by no means a measurement of performance or a tool for motivating or managing a team.

The metric, in fact, is informational, not motivational. Robert Austin brings some clarity to the difference in his book Measuring and Managing Performance in Organizations. A poorly used metric or one that’s employed in a situation that doesn’t call for it can quickly become a “bad metric.”

Having too many metrics in place can create noise and make it more difficult to zero in on the things that matter. So it’s crucial that companies select metrics that they absolutely need to understand their product development processes. The good metrics allow a company to diagnose its issues and create solutions to resolve them.

When you select a metric, you should ask yourself “So what?” In other words, if the metric you select reveals an issue, what will you do about it? If you can’t answer that question, the metric likely won’t do you any good. A good metric is one that tells you something useful without causing harm.

Bad metrics, on the other hand, have some pretty obvious faults. Bad metrics typically:

• Take more time and resources to collect and review than they add in value.
• Create perverse incentives, causing teams to behave in a way that’s harmful to the business.
• Diminish a team’s passion and morale, hurting its productivity and output quality.
• Are liable to inaccuracy or are easy to manipulate.
• Don’t tell you what you think they’re telling you.

You obviously want to avoid those bad metrics, but be wary of good metrics that can evolve into bad metrics when they’re misused. As I mentioned above, not all metrics are meant for all situations. The metric I discussed with the CTO might be applicable to his company, but not to all companies.

The key is knowing which metrics are useful for your company. As you weed out the bad and employ the good, it will inevitably improve your product development processes.