Quality Insight

Written by Dilpesh Bhhesania (IBM), and Richard Jordan (Nationwide Building Society).

Quality is not an action; it’s a habit. Aristotle said this to be true today in relation to a Product or Process. Quality is the foundation of all that an organisation strives for. Although quality can be subjective, having a way to measure the quality position is important. There are many reasons teams might choose to measure quality in the context of product delivery or application. Some of these are discussed here.

Traditional Waterfall delivery required that the product’s perceived quality was verified by a separate Testing group. This team would then design and execute a series test to verify the solution meets the organisation’s requirements. This approach raises the question of who is ultimately responsible for quality. However, the link here is the type of metrics that would have been used in order to evaluate the quality of a product or process.

Traditional delivery is judged by the number of defects found, either in totality or at a rate that they are detected. This should not be confused with Defect Density, which is a separate metric. The total number raised of defects is simply a numerical value. The lower the number, the less quality is perceived.

There are other areas that measure quality:

• Test Status – Passed/Failed/Blocked/No Run
• Test Schedule – Progress against Plan
• Planned vs. Actual

These metrics were presented in different flavours, dividing them between platforms, applications, teams, etc. Besides calling out top blockers, RAG states, and the path to green, Areas that are more mature in reporting could include metrics such the total effort and/or costs associated with testing, feature stability, etc.

These metrics were useful for showing progress in testing, but they did not show the quality of the product or processes. It also failed to provide a consistent, unified view of quality throughout the entire delivery lifecycle.

Another controversial view is that in traditional delivery structures, teams would produce metrics because this is what the Management told them to produce. Project managers would prefer to see numbers that were focused on the number of tests, with the view that more tests equals better quality and progress. It was difficult to show the value of Testing and, more importantly, Quality when metrics were not properly interpreted.

There is a common misconception that agile methods are automatically associated with more collaboration. This is not true, especially if an Organisation’s culture or mindset is traditional.

Quality is not confined to one area. This article aims to highlight some measures that can be used to show the quality of the product and embed it throughout the lifecycle. These metrics encourage engineers to take a Quality first approach, to actively look for areas where improvements can be made or to make efficiencies. Rather than defaulting to the Test function to use these metrics to show progress, they encourage them to adopt a “Quality first” approach to engineering.

It will also identify efficiency measures that can be used to complement the overall reporting structure. However, the key is to help people shift to the mindset that metrics are meant to provide valuable insight into products, processes, and teams. Blindly producing “throwaway” metrics simply because it’s been done that way is a hindrance in an organisation’s Quality transformation journey.

Why should teams measure – Purpose

There are many reasons teams might use certain metrics to track progress, efficiency, and quality. Take, for example:

• Evaluate Product Health – A quick visual and/or numerical indicator of the “healthiness” of your product can help you determine where to focus to correct or maintain your current trajectory.
• You can use these to determine progress and understand when a goal or story will be achieved. This also shows any obstacles a team might be facing that could hinder progress and ultimately affect the quality of a product.
• Requirements Coverage & Traceability – Quickly determine where there are gaps in coverage and areas of risk
• Uncover Inefficiencies – This can come from many perspectives, including code rework and value realization, time to market, time to market, etc. It is easier to identify areas that need remediation by having a set metrics that can show where inefficiencies exist throughout the product’s lifecycle.
• Articulate Product Complexity-You might ask, “Why should I care about Product Complexity?” Many organisations wouldn’t consider this a quality indicator or measure it. Poor product management can lead to poor quality products, such as bloated features, inaccuracy or poor design. This will not only lead to more rework and higher costs, but it will also reduce value. McCabe’s Cyclomatic Complexity helps you understand how complex a product might be. Complexity doesn’t necessarily mean that a product has to be complex. This metric can be used to help clarify unit test coverage.
• Adherence to Acceptance Criteria/Definition of Done
• Articulation of risk / Problem areas – Metrics can give valuable insight into the risks that may be present in both the product and the process. It is rare to see metrics that help to define risk positions, which can be used in other areas.
• You can create feedback loops by increasing the frequency of issuing metrics. This can either create more feedback loop instances or create one that has never existed.
• Investment Payoffs – Quantify the impact of certain strategies or investments on the quality of a product, or how a team’s performance.
• Automation effectiveness – This will help you determine whether your Automation strategy is helping to deliver your product(s), without compromising on quality.
• Accountability – Quality is everyone’s problem. Everyone should be held accountable. Metrics are not enough.
• Speed to market / Product release – End users want functionality quickly. This is a constant demand in today’s rapidly changing technology landscape. Having a look at how quickly features are released can give insight into whether your customer base is satisfied.
• Trend Analysis/Continuous Improvement – Determine quickly if improvements are paying off, or if quality has dropped. This view allows you to make earlier corrections and remediations.

DevOps Research & Assessment (DORA), Programme

The DORA was a research program that lasted seven years and looked at data points from more than 32,000 professionals worldwide in the IT industry. In 2018, Google acquired it. Google acquired it in 2018.

The DORA published “Accelerate”, a book in 2018. In it, the team identified a core group of metrics that could be used to show the capabilities of software developers and their delivery. These 4 metrics are designed to help teams make informed decisions based on data. They can improve their delivery rate, work practices, and product reliability. It was found that the top performers will strive to improve on these four metrics.

• Change Time This measures how long it takes for a team to have code running in production after it has been committed in development environments. This can indicate how mature or equal the delivery process, and also highlight inefficiencies. Lack of CI/CD pipelines and shared environments, isolated testing and development teams, and cumbersome Route to Live processes can all lead to longer lead times. Team members who aim to achieve “elite” status can have their code running in Production within a day. Others may have only monthly, quarterly, or even half-yearly deployments. This increases the risk that you are trying to deliver too many features, which could lead to poor user experience or system outages.
• Deployment Frequency This shows how often changes are made to Production. This can be used at any level, i.e. This can be used at any level of the organization. However, if there are teams with greater maturity in DevOps skills then these metrics can also be applied locally. In both cases, the underlying insight is still valid. Teams strive to make production changes that are smaller and more frequent, which will help speed up the realization of benefits for end users. This has the added benefit of reducing the chance of regression or production failures. Teams that strive for elite status can deploy almost immediately, while teams that have less maturity in DevOps may have only monthly, quarterly, or even half-yearly releases. It is easy to calculate the deployment frequency and use this information to help determine if there are any other impediments that may be preventing faster and more frequent production releases.
• Change failure rate This shows how many times a change is pushed into production and results in a failure. It could be complete outage, poor service or inaccessibility. It’s when a change enters Production and requires a fix. A team can measure the Change Failure Rate to assess its maturity in its deployment process. This could be either manual or integrated. This can be used to highlight quality issues that may have been overlooked earlier in the Product’s lifecycle. Quality features should always be the priority. Quantity. The organisation can set targets for failure rates based on risk appetite, but teams should aim to achieve near-zero failure rates. Near-zero failure rates will be achieved by more mature and “elite” teams due to their team culture, as well as the other metrics discussed in this article.
• Mean time to recovery (MTTR) This measures the reliability of your applications and systems. It measures the time it takes for an organization to recover after an incident or outage. Failures are inevitable, but it is important to be able to recover from them. While teams that aim for elite status can recover in minutes or hours, others may need to take weeks or months to recover. This information can provide valuable insight into other areas, such as whether an organisation has adequate alerting and monitoring capabilities, or if the number and size of Production deployments need to be reviewed. Iterative, smaller releases are better for users and reduces the chance of production failures.

These DORA metrics are used to show that there is no one way to measure quality, stability, or organisational maturity. However, they can be a useful starting point for organisations and teams when combined with other metrics.

What should teams measure – Quality & Efficiency Indicators

The table below outlines some of the metrics that can help to show the efficiency and quality of a product, team or organisation. The links are to one or more DORA metrics. Some metrics cannot or should link back to DORA. This is to show that four metrics will not provide comprehensive insight. These metrics are not intended to be prescriptive, but rather to shift the mindset from “We have to provide metrics because that’s what’s being asked” to “What are these metrics actually telling us about my organisation’s position on quality?”

Where should metrics be made available to teams?

Any platform used by teams can make metrics available. There is no set of rules; the metrics should be consistent. It is not a good idea to have different metrics reported in Jira and Confluence. This doesn’t make sense, and doesn’t provide all stakeholders with the same information. It’s all about efficiency.

Platforms like Confluence, Jira and Test Management tools are all possible to be used. To demonstrate certain capabilities, it might be beneficial to create reports using the dev ops chain toolset if more is being done in CI/CD.

Access to information is more important than ever. It’s essential to have the right tools to help you make informed decisions. Stakeholders don’t want to have to search in many places for the information they require.

Who should use these metrics?

These reports should be automated to take in input from all tools used to support the activities within delivery functions. This could include design, build, testing, infrastructure, pipeline activities and build releases. These allow the reporting of the above metrics. These metrics can be owned, for instance:

• Sprint Teams
• Scrum Masters
• Quality Engineers/SMEs/Leads
• Technology is the Key to Success
• Support Teams
• Incident Management
• Product owners

Reporting alone is not enough. These metrics must provide tangible and meaningful insights that allow teams to continuously improve their performance and to strive for faster, more agile and more iterative delivery. To recognize the areas that are performing well and to take responsibility for those areas that aren’t, it must be a coordinated effort from all parts of the delivery function.

When are these metrics to be collected?

There is no set rule regarding the frequency at which metrics should be collected. You can collect them as often as you feel necessary. This is because it allows for quick feedback and better quality. An automated reporting system would allow reports to be sent daily to various events, whether they are stand-ups or retrospectives. It is also useful in prioritizing backlogs and identifying key areas to focus on quality issues.

• Daily Report generated automatically by hourly metric collection (depending on velocity, team construct, etc.).
• Weekly Summary / End of Sprint Summary: Enabling trend analysis, continuous improvement
• Monthly – Meetings with key business/stakeholders, etc.

These metrics are not intended to be used as a guideline and should not be interpreted as such. It is clear there is no single metric which will provide holistic or complete insight into the quality of software/product/code. These metrics are intended to be used as an example of how an organization can include certain elements in their reporting, and more importantly, to provide meaningful insight into the Quality journey. These metrics are dependent on the production and consumption of the individual teams, but it is important to agree on consistent generation. You might consider tying metric generation to key delivery milestones or aligning it with sprints. The frequency of metrics is flexible as long as there’s value from them.

Your organisation may want to begin capturing metrics. However, it can be overwhelming to try to measure all of them. It may also make it difficult to see the forest for the trees. You will be able to gain more insight and influence greater change by starting with a subset of metrics that are relevant to your stakeholders. Consider this: What would you choose if you were just beginning to report metrics?