# AI code reviewers should provide executable evidence

> Why AI code reviewers should validate findings with executable and reproducible evidence.

Published: 2026-08-13T00:00:00+03:00
Canonical: https://ryanzidago.com/posts/ai-code-reviewers-should-provide-executable-evidence/

Code-review agents should provide evidence for actionable findings whenever possible.

A code review comment is a claim. A failing test, a comparison with the default branch, or a query against production data is a claim **and evidence that others can verify**.

This makes findings more reliable, easier to understand, and cheaper to communicate.

More reliable:
- Verifying findings against code, the default branch, or production data filters out false positives and distinguishes defects introduced by the current change from pre-existing behaviour.

Easier to understand:
- Evidence makes the conditions, observed behaviour, and expected behaviour explicit.

Cheaper to communicate:
- Concise evidence can replace a lengthy prose explanation, reducing review effort and context window usage for humans and AI agents alike.

I have concrete examples at work where Claude Opus 5 used 2,892 characters to explain something that Codex had expressed as a failing test in 20 lines of test code.

# Examples

## Validate a finding against code

When a finding claims that a particular input or state produces incorrect observable behaviour, prefer a failing test against the expected correct behaviour.

### Avoid

A long prose comment that asks the reviewer to reconstruct and verify the claim:

> `Order.total/1` appears to allow discounts greater than the subtotal. This can produce a negative total when, for example, an order with a subtotal of 100 cents receives a discount of 150 cents. The total should probably never be lower than zero. Please consider clamping the result or validating the discount before subtracting it.

The finding may be correct, but the reviewer still has to determine whether the input is valid, reproduce the issue, and confirm the current behaviour.

Also avoid writing a test that merely confirms the incorrect behaviour:

```elixir
test "the total is lower than zero" do
  order = %Order{subtotal: 100, discount: 150}

  assert Order.total(order) == -50
end
```

This proves the current behaviour but does not declare what the correct behaviour should be. The test will also become obsolete as soon as the defect is fixed.

### Prefer

Express the same finding as a failing test:

```elixir
@doc """
Applying a discount greater than the subtotal produces a negative order total.
A negative total could result in an invalid refund or credit instead of a charge.
"""
test "the total cannot be lower than zero" do
  order = %Order{subtotal: 100, discount: 150}

  assert Order.total(order) == 0
end
```

The test makes the input, disputed behaviour, and proposed expectation explicit. Running it proves whether the finding applies to the codebase: if the test fails because `Order.total/1` returns `-50`, the reviewer has executable evidence of the defect.

And if you want to make it even easier for your agents to communicate code review findings, tell them to follow these rules for writing [self-contained tests in Elixir](https://ryanzidago.com/posts/self-contained-tests-in-elixir/).

### Notes

Not every finding should be reproduced as a test. Findings about naming, readability, or architecture may be clearer in prose.

Some behavioural findings are also difficult to reproduce reliably, including concurrency bugs, migration failures, and interactions with third-party or external services that require extensive mocking. Be cautious about insisting on a failing test in these cases: an AI agent may spend 50 minutes adding substantial setup and complexity only to produce weak or misleading evidence. When the cost of a reliable reproduction is disproportionate, a clear explanation of the risk and the conditions under which it may occur is more useful.

## Validate a finding against the default branch

Before attributing a defect to a pull request, reproduce it against both the proposed change and the default branch.

### Avoid

Leave a blocking comment on the pull request without establishing whether the change introduced the behaviour:

> This input causes `Order.total/1` to return a negative value. This pull request should prevent that.

The finding may be valid, but it may describe a defect that already exists on the default branch and is unrelated to the proposed change.

### Prefer

Run the same reproduction against both branches and report the result:

> The test fails on this branch and on the default branch, so the defect predates this pull request. I opened a separate GitHub issue containing the reproduction and resolved this review comment.

If the finding reproduces only on the proposed branch, the agent has evidence that the pull request introduced it. If it also reproduces on the default branch, the agent should usually track it separately rather than presenting it as a regression caused by the pull request.

### Notes

A defect that also exists on the default branch can still be relevant to the pull request. The change may expose the defect to new inputs, increase its impact, or depend on fixing it. Comparing branches establishes whether the pull request introduced the behaviour; it does not decide by itself whether the finding should block the change.

## Validate a finding against production data

Code reviewers often raise concerns about unusual data that could theoretically exist. An agent can check production data before asking the pull request author to handle a speculative edge case.

### Avoid

Leave a comment based only on a hypothetical:

> What if an appointment has a duration of zero?

The reviewer has identified a possible edge case, but has not established whether the implicated data exists in production. The pull request author now has to investigate the claim or defend against a state that may never occur. This is particularly infuriating when the product has been live for years and has millions of records.

### Prefer

Look for production records that support the concern:

```sql
SELECT id, start_datetime, end_datetime
FROM appointments
WHERE start_datetime = end_datetime;
```

Then report what the evidence establishes:

> I queried the millions of appointments in production for one whose start and end times are equal and found no matching records. The available production data does not support the zero-duration edge case, so I resolved the comment.

Finding no matching records does not prove that the state is impossible or could never occur in the future. But it gives the agent evidence for deciding whether a theoretical concern is relevant to the current pull request instead of passing every hypothetical on to the author.

### Notes

The absence of matching records is evidence about the current dataset, not proof that the state is impossible. Historical data may have been deleted, a rare state may not have occurred yet, and application rules may change. The reverse is also true: production databases may contain test or demo data, so finding a matching record does not necessarily prove that the state occurs in real usage. Production queries should inform judgement rather than replace reasoning about what the system permits.