Measure Quality

How to discover quality issues and how to act on those

José Caldeira avatar
Written by José Caldeira
Updated over a week ago

Quality is a critical element of your development process. Whenever you are trying to accelerate you want to keep an eye on the final impact on Quality and ensure the process identifies impact of moving faster.

How to use Athenian to improve the quality process

Quality Dashboard

The Quality dashboard shows you the flow of bugs that have been worked on, and how much time was invested into working on these bugs.

Athenian classifies as a bug anything that has been tagged as such on Jira.

MTTR

The Mean Time to Restore (MTTR) metric tells you the average amount of time it takes between a bug report being filed, and its resolution.

MTTR = Bug Released - Bug Created

Bug Released = Latest event time between Pull Request release and Jira ticket moved to solved status

Bugs Raised vs. Fixed

The Bugs Raised vs. Fixed ratio will show you how many bugs are being fixed compared to how many new ones are being reported.

Bugs Raised By Priority

The Bugs Raised by Priority section is color-coded, with green representing low priority bugs, and red representing high priority bugs. The MTTR metric only considers bugs classed in the two highest priorities.

Resolution Dashboard

You can see the average time for each type of bug to get resolved, from the highest priority to the lowest. You can also view a timeline of the Mean Time to Restore, based on the priority of the bugs, which will show you how your teams’ MTTR changed across your specified time period.

Below this, you can compare the Mean Time to Acknowledge bugs to the Mean Time to Repair them, again based on priority.

Acknowledge means the average time elapsed from the bug creation until work began on the issue – the earliest timestamp between the transition to In Progress and the 1st commit of the associated Pull Request.

Repair means the average time it took to implement the fix, from the moment work began until the fix was released into production.

The Issues Table

The Issues Table view can show you the underlying data for your quality control processes.

If you filter the table by “type:bug”, or any other type, you can see data for every bug your team is currently working on. You can then filter this table by size, priority, and other metrics to get a more detailed view of how your teams manage bugs as part of their workflow.


🔎 What you to look for?

  • Is the bug fixing ratio increasing or decreasing?

  • What’s the MTTR? If it’s high, what’s making it so high?

  • What’s the distribution between time to Acknowledge and to Repair? Is this compliant with your SLOs?

  • What’s the average Pull Request Size to solve bugs? Are you increasing risk by releasing complex items?

🚀 Actions to improve

  • Ensure you have a healthy Bug Fixing Ratio – validate your investment time per category and ensure you increase your investment in bug fixing.

  • Automate – ensure you create regression testing to accelerate the release of the bug fixes.

  • Remember that a high number of P0 and P1 bugs, and a high MTTR, might not necessarily indicate that your team is dealing with numerous severe bugs. It may simply indicate that you have an issue with your prioritization system.

  • Discuss bug prioritization and acknowledgement with your teams, check if they are aware of the time to work on tickets and look for ways to automate the process and remove blockers - e.g. send a notification if a bug has stayed in the backlog for more than x days.

Did this answer your question?