Reviews and Audits

Reviews and audits are systematic examination techniques used to evaluate software products and processes for quality, correctness, and standards compliance. They represent the most cost-effective defect detection method available—studies by Fagan (IBM), Capers Jones, and others consistently demonstrate that formal reviews find 60-90% of defects at a fraction of the cost of testing. A defect found during a code review costs approximately $100 to fix; the same defect found during system testing costs $1,000; found in production, it costs $10,000 or more. Reviews catch defects earlier, closer to their source, when context is fresh and fixes are simple.

Types of Reviews

Informal Review (Buddy Check): A colleague reads your work and provides feedback. No formal process, no documentation, minimal overhead. Useful for quick feedback but lacks the rigor to ensure thorough examination.

Walkthrough: The author presents their work to a group of peers, explaining the logic and design decisions. The audience asks questions and identifies potential issues. Walkthroughs are somewhat structured but remain author-driven—the author controls the pace and focus.

Technical Review: A defined group of qualified reviewers examines a work product against specifications and standards. Unlike walkthroughs, technical reviews are reviewer-driven—reviewers prepare in advance and systematically check the artifact against defined criteria. A moderator ensures the process stays focused and productive.

Formal Inspection (Fagan Inspection): The most rigorous review type, following a strict process with defined roles, preparation requirements, and metrics collection. Inspections are the gold standard for defect detection but require significant investment in training and process discipline.

The Formal Inspection Process

Michael Fagan at IBM developed the inspection process in the 1970s. It remains highly effective:

Planning: The moderator selects reviewers, distributes materials, and schedules the meeting. Reviewers are chosen for relevant expertise—a database expert for schema reviews, a security specialist for authentication code.

Overview: The author provides context about the work product—its purpose, design decisions, and any known issues. This enables reviewers to prepare effectively.

Preparation: Each reviewer independently examines the material before the meeting, noting potential defects and questions. Preparation typically requires 1-2 hours per 200 lines of code.

Inspection Meeting: Reviewers meet (typically 1-2 hours maximum) to discuss findings. A reader paraphrases the work product line by line while reviewers raise issues. The recorder documents each defect found, classified by severity and type. Critical rule: the meeting identifies defects but does not solve them—solutions are deferred to rework.

Rework: The author addresses each identified defect.

Follow-up: The moderator verifies all defects have been adequately resolved. If many defects were found, a re-inspection may be warranted.

Roles in Formal Inspections

Real-World Example: Code Review in a Financial System

A trading platform team conducts code reviews for every change:

Pull Request Review Process:

1. Developer creates a pull request with description of changes and linked requirements
2. Automated checks run (linting, tests, security scanning, coverage)
3. Two reviewers are assigned (one domain expert, one general reviewer)
4. Each reviewer has 24 hours to provide feedback
5. Reviewer checklist includes: correctness, security implications, performance impact, test coverage, documentation, error handling
6. All comments must be resolved (either fixed or explicitly acknowledged with justification)
7. Both reviewers must approve before merge

Example findings from a single review:

• Race condition in concurrent order processing (Critical—would cause duplicate trades)
• Missing null check on market data response (High—would crash during market outages)
• Inefficient database query scanning full table instead of using index (Medium—performance degradation)
• Inconsistent error message formatting (Low—cosmetic)
• Missing unit test for boundary case (Medium—test gap)

Without this review, the race condition might have reached production, potentially causing millions in erroneous trades.

Software Audits

Audits differ from reviews in their focus—audits assess process compliance rather than product quality:

Process Audit: Verifies that the team is following defined processes. Are code reviews actually being conducted? Are test cases traced to requirements? Is the change management process being followed? Process audits sample evidence of compliance.

Product Audit: Verifies that deliverables conform to their specifications and standards. Does the documentation match the templates? Does the code follow coding standards? Are all required artifacts present?

Configuration Audit: Verifies that the software configuration (code, documents, data) is consistent and complete. Do the deployed binaries match the source code? Are all approved changes—and only approved changes—included in the release?

Metrics from Reviews

Reviews generate valuable quality data:

• Defect Detection Rate: Defects found per hour of review time (typical: 8-15 defects/hour for code inspections)
• Defect Density: Defects per KLOC found in reviews (indicates code quality before testing)
• Preparation Rate: Pages or lines reviewed per hour during preparation (if too fast, reviewers are not being thorough)
• Review Coverage: Percentage of code or documents that undergo review

Making Reviews Effective

Keep sessions short: Reviews longer than 2 hours produce diminishing returns as attention degrades. Better to review less material thoroughly than more material superficially.

Review small changes: Reviewing 200-400 lines at a time is optimal. Beyond 500 lines, reviewer effectiveness drops dramatically—the brain cannot maintain focus over large volumes.

Use checklists: Common defect categories (security, performance, error handling, concurrency) serve as reminders for reviewers, ensuring systematic coverage.

Foster psychological safety: Reviews must be collaborative, not adversarial. The goal is finding defects, not judging developers. Teams that fear reviews hide code from scrutiny, defeating the purpose entirely.

Track and improve: Collect data on what types of defects reviews find. If certain categories consistently escape reviews, update checklists or provide targeted training.