How to benefit from process metrics?

There has been a lot of talk about the importance of data in running a successful business (e.g. the great book Lean Analytics by Alistair Croll and Benjamin Yoskovitz). Evidence-Based Management has also gained recognition in the Agile world with the EBM Guide, published in 2020, describing how this approach and method might complement other Agile tools.

Although Evidence-Based Management focuses globally on improving “customer outcomes, organizational capabilities and business results (source)”, one thing is certain - measuring the right things (whether product-wise or process-wise) and analyzing them is the best way to make informed decisions and plan experiments which help us improve.

Why data and process metrics are important?

Why do we need data and process metrics in the first place? Isn’t it troublesome to track something? If you believe in empiricism as understood in Scrum: “working in a fact-based, experience-based and evidence-based manner” (source), you cannot base your opinions and decisions on mere assumptions or gut feeling. We actually need transparency, inspection and adaptation to know what the current state of events is in order to influence it in the best way possible. That is precisely where metrics come in handy.

Choosing the right business process metrics

When it comes to business process metrics, many people ask, “Which metrics should I implement in my project?” and they start searching for the most popular ones on Google. “What is the most optimal set of metrics I can use?” And since, more often than not, we believe “the more, the better,” we search for as many metrics as we can to prove that we’re data-obsessed and following the current trend. In reality, even though data and process metrics are very important and can help us (and our products) succeed, there isn’t a one-size-fits-all solution or recipe that will guarantee success for everyone. Why is that?

That is because, like Simon Sinek once said: “we should always start with why.” Since metrics should provide us with data and evidence on which to base our decisions, there is no value in tracking something that will never influence our behavior or is simply not relevant to us.

We need to understand which aspects of our work contribute to our vision and our definition of success. For some teams this will be the quality of their work, for others the speed with which they deliver value to their customers. It may also be a mix of the two or something completely different. Once we have figured it out, we can start thinking about, “how will we know that what is important to us is at a satisfactory level?” Some teams will have to consider how they will know if what they produce is of good quality. We need to determine the indicators.

How to tell which business process metrics are important?

Depending on the circumstances, for some teams the right indicator might be the number of bugs that they produce, test coverage, failure rates, etc. As you can see, the options are numerous and are only limited to what we consider valuable. Having determined what indicators we would like to use, we need to ask ourselves a question: “what value for this indicator would be our goal, our dream value?” Is it going to be less than two bugs per month, 90% test coverage, or something else completely?

With this desired value in mind, we can start tracking the current value of the chosen process metric and regularly plan actions or experiments that might push us towards the desired outcome. This can (and should) happen during every retrospective, giving us a fantastic opportunity to inspect and adapt in small steps.

You might be also interested in the article:

How to improve productivity in agile scrum teams

Tracking process metrics - which ones really matter?

If process metrics should be tailor-made for every team depending on their needs, priorities and circumstances, why is it that the majority of teams usually track the same “basic metrics”, like velocity, predictability, burndown? This is because, for many teams at least, the “basic” level of speed, quality and flow is still important.

I haven’t met a team yet which would value speed of delivery while completely disregarding the quality aspect. This team would probably develop software quickly (usually to the delight of the management) but their product would probably not be usable. The same goes the other way around - if we were to focus exclusively on quality - providing only products of the highest imaginable quality but doing it so rarely that the customers would see no value in it - this wouldn’t be desirable either.

For that reason, even if you’ve determined the most important aspects of your processes and workflow that you want to track in your team (and created custom-made metrics for them) you should always consider to what extent the other aspects, deemed ‘less important’, should still be tracked, or at least acknowledged.

An examples of process metrics

At Boldare, we have a pre-selected set of digital product metrics that we use at particular phases of the product development cycle because there are things that we consider important, regardless of the priorities that we set on top of them. These metrics include:

• sprint goals achieved: Sprint goals are basically the essence of value that we want to deliver during the sprint and therefore they become the most basic indicator of the scrum team’s effectiveness. If we notoriously fail to deliver the value we wanted to and don’t achieve the sprint goals, then most probably something is blocking us, our workflow is not optimal, we’re lacking decisiveness, competencies or knowledge, etc. Sprint goals act as a kind of litmus paper that tells us that there is a problem somewhere that we should explore and find a solution to.
• velocity: We track velocity to be able to estimate the time of delivery/release and adjust our work mode (or scope) to changing priorities, reality and current circumstances. It is, however, very important to remember that with velocity, the ultimate goal should never be to simply work faster and faster, as this is usually to the detriment of other important aspects of the project.
• predictability: We track the ratio of “done” items versus “planned” ones, among others, so that we know if our refinement process (and the resulting understanding of the acceptance criteria and business requirements) is good enough for us to properly plan the sprint. Of course things happen during the sprint and achieving 100% predictability all the time might not be possible - but if the metric shows that we are at about 50% most of the time, it is a clear indicator that we need to have a closer look at the topic and look for gaps in the process.
• team morale: We track this so that we know whether our teams are happy and comfortable in their daily work. This metric is very important to us as we know that a comfortable team translates into better quality, better ideas, and even more speed (which, ironically, when being pushed too much might backfire, resulting in lower team morale and consequently an even lower speed)
• client satisfaction: We track this to check if the client is satisfied not only with the outputs of our work but also with our way of working, communication, transparency, and all the other things that influence their overall level of satisfaction.

All of the above digital product metrics work as a “basis”, as a first beacon suggesting that we may be approaching some perilous shores but they are usually not enough to guide us to safety. That is where the custom-made metrics mentioned before come to play.

You might be also interested in the article:

Read also

Product Strategist - a role that transforms digital companies

Process metrics at Boldare - our example

Some time ago, one of my teams had a problem with achieving sprint goals and were demonstrating low predictability. After some investigation, it turned out that the main culprit was adding “unplanned but very urgent” tasks to an already running sprint. It’s not surprising that when you add more work to a sprint that has already been planned to be “full”, you lose focus and do not have enough time to finish everything that was originally planned.

That is why we introduced a new process metric showing us how much time we spend on unplanned issues every sprint. The number was very high and we wanted to lower it (or even eliminate it altogether). In order to do that, during every retrospective we talked about the issue, proposed different solutions we might try to avoid unplanned work, and monitored the metric for improvement.

Step by step, by inspecting what kind of issues dropped in “unannounced”, where they came from, why it was that they couldn’t wait until the next sprint, etc. we managed to change different elements of the process and solve the issue.

Once the issue was solved and the metric was showing very little time spent on unplanned issues, we stopped tracking the metric and deleted it on the principle that, “you should only track what is currently important to you.” Since the problem no longer existed, there was no need to keep maintaining this particular business process metric.

Process metrics - a summary

This example vividly illustrates the rule that, in my opinion, should be followed whenever we use metrics: process metrics should be constantly inspected and adapted, they should be alive in order to give us real value. They should be embraced, not feared or neglected, and used not only because they are fashionable but because they are a very basic and simple but powerful tool in any Agile environment and can make a great difference to the success of your products.