How much manual testing do you do when deploying code?

Determinants of the Extent of Manual Testing Needed in a Codebase

The extent of manual testing needed is not a static figure, and it varies depending on the project’s nature, requirements, and the industry it belongs to. Here are a few factors to consider before deciding the amount of manual vs automated testing in your codebase:

Project Complexity and Size

These parameters largely influence the time, effort, and resources dedicated to both manual and automated testing. Complex and large projects require a multifaceted approach to manual testing, covering multiple scenarios, integrations, user roles, and edge cases, which can be resource-intensive and time-consuming. This calls for more manual and automated testing to ensure the software works as expected. For example, a banking app that offers many services like online banking, investments, and insurance is very complex, with many features, security rules, and user roles. This means it needs detailed manual testing to ensure every part works well together and meets strict regulatory requirements imposed on financial systems.

More extensive and complex projects usually demand a more rigorous manual testing approach. In contrast, small, straightforward projects require less manual testing as most tests can be handled automatically.

Industry Regulations

For industries like healthcare or finance, which governments heavily regulate, comprehensive manual testing is often unavoidable to meet compliance standards. In such a case, all the key areas highlighted by the regulatory body should be manually tested to ensure the software passes approval by the government.

Potential Risk

The components and features identified as high-risk areas in an application must be tested manually to ensure bugs are resolved. High-risk areas in an application are usually those that are more prone to errors or where a mistake can lead to severe consequences.

These areas are identifiable based on several factors, such as the complexity of the code, historical data of bugs, the criticality of the feature, and potential security vulnerabilities. Continuing our example on a banking application, a feature handling monetary transactions would typically be high-risk due to its sensitivity and the potential impact of errors. This feature should include comprehensive manual testing since a single mistake can cause a considerable loss in revenue.

Software Development Context

Despite the high reliance on automated testing today, the emphasis on manual testing in specific contexts is unavoidable. For instance, when dealing with graphics-intensive applications, manual testing is necessary to verify rendering and quality of renders. This is very difficult to automate. In the same light, different projects, such as web applications, mobile apps, and embedded systems, demand distinct manual testing approaches due to their unique interaction modes and operating environments. For instance, web applications require cross-browser and accessibility testing, while mobile apps need testing across different devices and operating systems, focusing on user interactions like taps and swipes. Embedded systems, on the other hand, emphasize hardware-software integration and real-time response testing.

In all these cases, manual testing is unavoidable, and resources and time should be provisioned, keeping in mind that manual testing of these critical areas has to be done.

Budget and Time Constraints.

The whole point of automation, besides taking repetitive tasks from developers and QA personnel, is to cut the budget and time spent between project development and rollout. The availability of resources and project deadlines can significantly influence the extent of manual testing performed. If you have time and budget, manual testing might be of great value; however, automating most of the tests except for the critical sections of the code is recommended when on a tight budget.

Enhancements and User Feedback

Post-release user feedback is critical for enhancing software quality and user satisfaction. It uncovers the real-world issues and improvement areas that might have been missed during the pre-release testing phases. Developers and QA teams must prioritize this feedback to make necessary adjustments and enhancements in subsequent releases.

Strategies for Deployment

When deploying code, adopting strategies like blue/green deployments, staging environment tests, and feature toggles followed by manual cursory checks in production can mitigate potential issues during deployment. Such techniques isolate problems that may arise during deployment and ensure seamless release processes. All these strategies are inherently subjective and manual.

Conclusion

So, to answer the question “How much manual testing do you do when deploying code?” the ideal approach is a balanced combination of both automated and manual testing that adapts to the project’s specific needs and available resources. While automation is integral in modern software development, manual testing remains fundamental as it fills the gaps left by automation, addressing key aspects like user experience, device accessibility, government regulations, and real-world applicability.