Right from the start, Melanie Ziegler knew she wanted to work in engineering. Straight out of college, she began her career journey, moving rapidly through the ranks of innovative engineering teams, first at Texas Instruments and soon thereafter at expansion stage startup Merit Technology in Dallas. Later on, she took on VP roles with a number of software companies including Advanced Visual Systems and VFA at which she led engineering and product development. In 2011, she brought her expansive experience to a new audience when she launched her own consulting firm, MSZ Consulting, LLC, to provide high-impact advisory and leadership services to early-stage and expansion-stage software companies.
The latest in Ziegler’s long list of impressive accomplishments is founding VPE Forum to provide Engineering VPs & CTOs at rapidly growing software companies with a high-caliber peer environment where they can collaborate on solutions and share best practices. “I saw early on that CEOs gained many benefits from having the opportunity to participate in peer groups,” Ziegler says of the inspiration behind VPE Forum. “I felt there was an opportunity to help engineering VPs by providing a trusted and confidential environment where they could get honest insights and advice into how to solve the very real business issues and challenges they were facing.” Since its launch in 2014, VPE Forum has been thriving and consistently helping executive engineering leaders learn and evolve so that they can make important contributions that help the companies they work for grow faster and achieve greater success.
One topic that is continually raised by Engineering leaders Ziegler works with in the VPE Forum is how to overcome the challenge of crafting relevant and useful engineering metrics that the business can use to evaluate success.
The Case Against Productivity Metrics
“Everybody struggles with this,” says Ziegler. CEOs often ask their engineering leaders to report productivity metrics in order to assess Engineering’s contribution to the business. “At the same time, engineering leaders know they can’t accurately measure productivity, their teams know it, and even if they could, they know it’s largely unrelated to the contribution engineering is making to the business.” “One of the main challenges I see is around what I would call activity-based metrics – which, at the end of the day don’t tie back to what’s really important for the business.” The problem stems, Ziegler explains, from the fact that it’s nearly impossible to measure engineering productivity. It’s an age old problem. Years ago the industry established a precedent where VPEs would report on lines of code (LOC) as a performance measurement. Thankfully, the industry has evolved beyond that simplistic and irrelevant metric, but vestiges of that mentality persist. Today, instead of asking for LOC, executives might ask engineering leaders to report on Agile metrics like velocity and story points to measure how much value has been delivered.
All too often engineering leaders deliver these metrics none the less because “Story points and velocity metrics are easy to produce,” Ziegler says, “but they are not terribly meaningful in terms of measuring team productivity. And they convey little about delivered business value. With regards to productivity, the problem is that things shift. The team’s skills make-up, size, engineering consistency, and availability are not constant within a team, never mind across teams. Vacations happen.” The benefit engineering organizations can bring is not about how many story points they delivered or their velocity, it’s about how agile they can be in meeting the changing needs of the business and the customer.
Engineering Metrics that Make Sense
So where should the focus be if it’s not on productivity metrics?
Ziegler says the focus should instead be on business outcomes. “Frankly, the types of metrics CEOs should focus on for engineering are the same types they use to measure the success of other functional areas, be it marketing, sales, or finance – business outcome metrics.” says Ziegler. After all, she points out, “You don’t grow a business on increased activity. You don’t generate revenue by selling story points. You grow a business on outcomes, such as new features delivered that are valued by your customers.”
To determine the most appropriate and meaningful metrics, CEOs and engineering leaders should collaborate to identify and articulate what the business actually cares about that engineering can impact. Is it subscription renewals, international support, features for a new vertical market? Understand the business goals and then define the metrics accordingly. Consider tagging stories in the backlog to the business impact they are intended to serve to make it more visible to all involved.
“Story points delivered do not equate to economic value,” Ziegler says. “Features delivered to enable entry into a new vertical market? That delivers economic value. That’s business impact.”
Another important element of developing sound engineering metrics is to ensure that there’s a strong collaboration between the engineering team and the product team. “Truly great delivered product is the result of a team, a true partnership between product and engineering,” Ziegler explains. “It follows that great metrics are derived from a similar partnership.”
Another challenge lies in balancing the often opposing goals of your development and operations teams. While developers are focused on deploying as much business-enhancing code as possible, the operations team is focused on maximizing up-time, and – conversely – minimizing the risk of breakage and failure that can accompany deployment. To close the gap between these two groups, you have to ensure that both teams are working toward a shared set of outcomes, outcomes that are based on calculated risk and which serve a shared audience and support the overall business strategy.
While Ziegler advocates strongly for this unified and outcome-driven approach, she doesn’t suggest that VPEs throw activity-based metrics out the window. Activity-based metrics are, she says, useful in managing and leading engineering groups. Measuring the team’s velocity, for instance, helps the team determine how much they can take on in an upcoming sprint. The key is to tailor the use of these metrics on a team-by-team basis. “All teams are not created equal,” Ziegler explains. “One team might need to focus more on improving quality, while another might really need to focus on turning up the dial on more effective user interfaces. The metrics you put in place for each of those teams are different.”
Even as she acknowledges the benefit of using activity-based metrics to manage workflow and assess team performance, Ziegler warns against letting those metrics find their way into other parts of the company. “What we don’t want to see as an engineer,” she says, “is the executive leadership looking to measure the engineering organization based on the activity-based goals that the VPE uses to plan the team’s work and drive continual improvement.”
To provide a different perspective – to the engineering team as well as to executives – she recommends evaluating the engineering team’s performance using “big,” outcome-focused questions like:
- Are you delivering value for the customer?
- Are you delivering results each quarter that will help grow the business?
- Are you mitigating risk?
- Do you know the ROI of the features you’re building?
- Did you ship when you said you would ship?
These are the kinds of questions that get the engineering team thinking in the context of what their work means to the business.
Examples of Outcome-based Metrics
So, while activity-based metrics have their place, it’s not to help the C-suite measure engineering performance. What kinds of metrics, then, help the engineering team report on productivity and quantify the value they deliver to the organization? Ziegler has several suggestions:
“One of the best practices I see amongst the collaboration between engineering and product is around defining the metrics that will be used to evaluate if a feature is successful, is being adopted, as was anticipated. After all, wasn’t there an ROI anticipated for the feature?,” Ziegler says. “Before the feature is developed, they identify the measures that define what success will look like for that feature.” In addition to putting focus on the outcome, this future-looking approach also gives the team the ability to engineer in a way to measure its usage. It gets them thinking about who is going to use the feature and for what benefit, about how the customer will define the success of the feature. Thinking about results in these terms helps eliminate situations in which a team has worked hard to develop ten new features, but no one cares: a situation that would demoralize the engineering team. Ziegler says “the resultant data also has the benefit of providing functional areas across the company with the insight they need to maximize the business benefit after initial release – whether it’s additional development that is required, more training or promotion, or any number of other paths.”
While Ziegler acknowledges that scorecards can be more qualitative than quantitative, she does see their value when the questions asked tie back to business goals. “Some companies in the VPE Forum use a product scorecard to review the business outcomes of the engineering organization,” she explains. “By having stakeholders from different parts of the company such as Customer Support, Sales, and Professional Services provide inputs, you put emphasis on the results of what has been developed.” Questions that serve well in this capacity focus on things like how easy the feature is to implement, how easy is it to support, how competitive it is, how is the quality, how satisfied are customers, and how complete is it compared to where it needs to be.
Cycle time measures from the time work starts until the time it’s finished, and Ziegler finds it a great measure of how quickly an engineering team delivers value to the customer. “I don’t really view it as a velocity metric,” she says ”it does measure velocity, but instead of looking only at speed, Ziegler looks at speed in the context of customer value. For instance, if you’re building out a new vertical area for the product, or if you need to work on stats for how quickly you can get something back to the customer – those are the types of things to look at in order to evaluate engineering performance.
The Lead time metric measures from the time a feature is added to the backlog to the time when the engineering team has the capacity to start working on it. It’s essentially how long the feature is aging in the backlog. “In addition to addressing the organization that feels engineering never gets to anything, measuring lead time can provide a lot of insight into engineering team capacity and other ways to attend to a roadmap,” Ziegler says. There’s value, too, in identifying features and functionality that have aged out because that can help the team recognize misaligned priorities. “Looking at lead time can provide insights that generate executive conversations about how investments are being made in the company and what’s really important,” Ziegler says.
Whichever metrics you use, make sure you consider the energy you’ll need to expend tracking them. Can the data you need be collected easily? Make sure that the return you’re getting warrants the effort. And even more importantly, make sure that the metrics you choose ultimately tie back to growth.
As Ziegler rightly points out, “If it’s not about increasing the revenue, achieving high customer satisfaction and growing the business, then why track it?”
“If you’re an engineering leader, my advice to you is think about the business strategy and make sure you develop metrics that tie back to that,” Ziegler says. “If you do that, it will crystallize your thinking around the decisions you make each and every day with your team. It engages your team and visibly ties their efforts directly to the company’s success.” Ziegler also recommends that the VPE market the engineering team to the rest of the organization. “Engineering simply can’t be heads down in their group, doing all the right things for the business, but not letting the business know about it,” she says. “The more visibility and sharing you have, the more the rest of the organization will know that engineering is producing really great things for the company and the customer.” Whether it’s in a newsletter, company meeting or an internal promotional video about a new feature, Ziegler pushes the teams she works with to establish a presence of success within the organization and a palpable sense of things getting done.
“When the company is happy with delivered results, with the business outcomes” she says, “the CEO’s first question to engineering won’t be about their productivity metrics.”