What is a reasonable code coverage % for unit tests (and why)?

Code coverage is one of the many software testing metrics that assist in assessing the performance and quality aspects of software in development and production. At a high level, code coverage percentage describes the percentage of software lines of code executed during testing. Check out our article on Code Coverage and Code Coverage Metrics.

Code coverage is one of the most controversial topics regarding software development, often resulting in heated debates about the actual code coverage percentage that standard software should achieve. The problem with these discussions is that they miss entirely the point of code coverage testing, which is creating quality bug-free code. Most documentation will tell you to achieve a certain percentage of code coverage based on some arbitrary figure. However, in this article, let’s explore the best practices that should drive code coverage percentage for unit tests and debunk some misconceptions.

Factors That Determine Code Coverage Percentage

Type of Software Project

Before creating a static number for the code coverage percentage developers should meet, consider the project. What are the goals of the code you are writing? How complex is your code? What are the most crucial paths and components of your software? Answering these questions will allow a developer to make better decisions on what percentage it should be. Write unit tests for all the critical paths of your software first and measure code coverage percentage. Critical path code coverage ensures that these identified crucial components are extensively tested and validated against the specified requirements. That way, you have a baseline to understand the best percentage of code coverage to achieve.

Unit Testing Quality Over Quantity

While having more tests may benefit a software project, the quality of those tests is often more crucial than the quantity. High-quality tests are meant to be comprehensive, maintainable, and meaningful. They target the critical areas of the codebase, are well-structured, and are designed to capture a broad spectrum of potential defects, edge cases, and failure modes. Check Best Practices for Writing Unit Tests for more details.

Focusing solely on quantity may lead to redundant, superficial, or ineffective tests, offering an impression of safety while leaving vulnerabilities unaddressed. In contrast, prioritizing quality ensures that each test serves a purpose, exposing defects and verifying that the system behaves as expected under various conditions.

The Law of Diminishing Returns

Aiming for a high code coverage percentage for unit tests is essential. However, one critical idea to remember is that the benefits gained by increasing unit tests may decline beyond a certain level, and the costs and efforts of writing such tests continue to increase. When a codebase has low to moderate test coverage, each additional test can significantly enhance the software’s reliability and healthiness by catching untested scenarios and reducing the risk of defects. However, as the test coverage approaches higher percentages, the remaining untested code often includes error handling or less critical paths with a lower probability of containing impactful defects. Writing tests for these parts can be time-consuming and complex and may offer little evident value regarding defect reduction or improved software quality.

Code Maintenance Practices

One of the most critical purposes of code coverage is that it is essentially a measure of what is not tested rather than what is tested.

In that light, incorporating code coverage in the code review process is more valuable than getting hung up on a fixed percentage of code coverage. During code reviews, let there be discussions on why some lines are not tested and why—this way, an optimal percentage of code coverage will be achieved. Ultimately, make sure that frequently changing code is covered by unit tests and that there is observability by the developers into what has not been tested. This ensures that tests are not getting worse over time.

Software Capacity

Another critical factor to consider is the software project’s time, budget, and skills capacity. If there are severe time constraints on the project, it is recommended to unit test the most critical parts first rather than aiming for a static code coverage percentage. Moreover, it is unrealistic to expect quality testing with a team that is unfamiliar with testing. Such a team may use code coverage as a tickbox they need to achieve, making them write crappy tests.

Conclusion

In conclusion, no one-size-fits-all “perfect code coverage percentage” can be universally applied across all projects. The extent of testing required for a codebase should be determined based on the business importance and critical nature of the code, the frequency with which the code will be modified or updated, and considerations like the expected lifespan of the code, its complexity, and specific domain-related factors. It’s impractical to dictate that every team should achieve a specific percentage of code coverage; the product owners best make such decisions with specialized domain knowledge. However, it is generally agreed in the industry that each software project should aim for a code coverage percentage of between 60% and 90%.

Any requirement to achieve a certain code coverage percentage should also involve investments in infrastructure to facilitate easy testing, including integrating suitable tools into the developers’ workflow. Additionally, it’s essential to be aware that setting a coverage target might lead engineers to view it as just another item to tick off, potentially neglecting further beneficial increases in coverage beyond the set target.