If you’re implementing Salesforce DevOps, then measuring performance to check the impact it’s having should be high on your to-do list.
We’ve previously discussed how measuring performance using Google’s DORA metrics can help you understand what progress your team has made on their journey of Salesforce DevOps adoption.
By using DORA metrics to measure performance, you can easily demonstrate the value DevOps is bringing to your business. Combined with your team’s cultural progression, this allows you to demonstrate the overall benefit Salesforce DevOps is delivering, whether that’s collaboration, agility, velocity, resilience, or process improvements.
But DORA metrics have their limitations. While they provide an overall idea of performance, they’re generic by design. As a result, a reliance purely on standard DORA metrics could cause issues further down the line as your DevOps processes mature.
According to a report by Gartner®, “DevOps Research and Assessment metrics, which are the de facto industry standard for DevOps benchmarking, provide insufficient visibility into the key organization-enabling factors required for success”.
What this means is that another layer of measurement, specific to your organization and business priorities, is required on top of DORA metrics.
Gartner recommends that tech leaders “connect and support business value creation by collaborating with key stakeholders to measure in their own terms, using an intuitive approach".
So how do you do that?
Gartner has laid out a 7-step process to help you understand what, where and how to benchmark and measure DevOps performance within your business.
A 7-step process to measure performance
Step 1 Identify your value-based goals with key DevOp stakeholders across the business. The goals should be directly aligned with the business, rather than generic indicators.
This alignment should cover market position, business economics, customer affinity and company strategy.
Step 2 Use a “dynamic capabilities” model to help determine the additional value DevOps is bringing to the business. This approach enables success by integrating, building and reconfiguring “internal and external competences to address changing business environments”.
Gartner explains how this can be “realized through an integrated set of capabilities":
- Sense new opportunities and threats
- Seize the moment, and respond with new products and services and ultimately
- Transform the organization through reconfiguring key organizational elements
Step 3 Work with key stakeholders to determine “enabling capabilities” of the top level performance metrics. You should also assess their maturity and also what impact they have across the whole organization. By only focusing on the primary metrics, it’s possible to misinterpret the underlying reasons for any poor performance in these areas.
The areas of primary enabling capabilities need to cover people, process, information and technology. Below those, the subcapabilities can include many things, including:
- Autonomy
- Collaboration
- Usability
- Extensibility
- Adaptability
- Measurability
The important thing to focus on is what is right for the team and stakeholders. As Gartner notes:
“The informational aspects must be designed with stakeholders in mind — not just the engineering-oriented DevOps team. Although it’s important to be able to measure team capabilities, these (and other) metrics need to have significance with the stakeholders. As a consequence, we believe that the development of the information-gathering capabilities needs to be a first-order priority, “designed into” the DevOps effort at the beginning.”
Step 4 Much like our recommended approach to implementing DORA metrics, this step requires you to assess the maturity of each capability within your organization, and how well you are currently doing at delivering them.
Gartner advises; “There will be little in the way of real-life data to support any specific rating, so the I&O leader will need to estimate the maturity in collaboration with the engineering team".
The main goal with this approach to benchmarking isn’t precision, but to develop consensus within the business on where opportunities for the biggest improvements lie.
Step 5 Look for any interdependencies across the subcapabilities you’ve identified. For example, an adaptable approach to delivering processes requires autonomy among your teams so they are able to adjust their workloads accordingly.
If you’re attempting to assess the impact of DevOps on one capability, without considering any interdependencies it may have with another, you may end up with the wrong understanding of how effective it’s been.
Step 6 Take all of these capabilities and subcapabilities, and agree what needs to be measured to assess performance in each. As these areas will be specific to your organization and dependent on other factors, Gartner doesn’t recommend any specific indicators to measure. Deciding what to measure should begin with putting together the important things you’ve assessed into one agreement that the whole business is on board with.
This should give your output metrics for each category so that you can begin to more accurately measure DevOps performance across the business, as shown in this example.
But as Gartner indicates, it’s also important to be aware of what may be missed with these measurements, and also when the best time to collect performance data.
“Many of the traditional DevOps metrics require an understanding of the potential drawbacks of the measure, as well as when it’s appropriate from the DevOps team’s perspective of the phase of the project in which they should be collected.”
Step 7 In this final stage, Gartner recommends that you combine your organization-specific performance indicators, align them with the standard DORA metrics, and then determine the best point in the process to collect measurements.
“I&O leaders just beginning their projects should not select metrics (or metric level) that would be unachievable for less-experienced organizations. In addition, there is no single way to implement a DevOps program, so I&O leaders should be focused on the goals relevant to their firm in a manner similar to the Agile Fluency Model. There is a practicality perspective concerning what to measure when as it is generally advisable to focus on the so-called stability metrics of Change Failure Rate and Mean Time to Restore Service first. Speed without quality will, of course, result in suboptimal outcomes.”
Next steps
Trying to measure DevOps performance without considering the individual requirements and structure of your business will result in suboptimal performance results.
Gartner recommends focusing on consensus, rather than collection, and becoming comfortable with estimation over precision, before you begin to collect and measure outcomes.
“Most importantly, start now and adopt the agile and DevOps principles of continuous improvement.”
Gartner, Data-Driven DevOps: How to Rethink the Measurement Approach, Published 29 March 2023, Cameron Haight and George Spafford.
GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.
