PR 337469 in microsoft/vscode got two Copilot reviews 52 minutes apart, on the same commit (352dd22d). The first said "Needs a closer look" and described four unresolved moderate issues. The second said three issues remained. Both bodies ended with the same structured line: Findings: None.
So on one unchanged diff, the summary text changed, the issue count changed, and the structured findings field reported nothing either time. I went looking for how common that is, and the answer is that the verdict is not part of the review's state at all.
A review object has a state, and Copilot's is always COMMENTED
GitHub's REST docs for pull request reviews describe a review as comments "grouped together with a state and optional body comment". That state field is what branch protection reads. Copilot code review writes COMMENTED.
I sampled merged PRs from microsoft/vscode in two slices of 30 (state=closed&sort=created&direction=desc), filtered to merged_at != null, then pulled every review object per PR and filtered to user.login == copilot-pull-request-reviewer[bot]. 28 PRs in the first slice and 27 in the second had a Copilot review. Every review object across all 55 was state COMMENTED. Zero APPROVED, zero CHANGES_REQUESTED.
That matters because of what reacts to state. GitHub's protected branch docs describe the merge failure you get when a review sits in a blocking state (GH006: Protected branch update failed... Changes have been requested), and the optional setting that dismisses stale approvals when the diff changes after approval. Neither path touches a COMMENTED review. A review that is never blocking cannot block, and it cannot go stale either, because there was never an approval to invalidate.
The heading is UI text. GitHub's code review concept docs say Copilot's reviews do not count toward required approvals by default, and that enabling Copilot approvals is a separate repository, organization, and enterprise setting. When that setting is on, a new push dismisses the approval. When it is off, the "Approval recommended" you are reading satisfies nothing.
Half the merged PRs carried a "Changes recommended" at merge time
Final Copilot review per PR, first slice, 28 merged PRs:
- 16 Approval recommended
- 9 Changes recommended
- 3 Needs a closer look
12 of those 28 merged with a non-approval verdict standing. The older slice (n=27) came out 14 Changes recommended, 12 Approval recommended, 1 Needs a closer look, with 15 merging on a non-approval verdict. Roughly half in both.
None of that is a bug report. A reviewer that informs without gating is a defensible design, and teams treating it as advisory are getting what the product promises. The problem shows up when you turn it into a metric. Count "AI approvals" or "AI blocks" from the review state and you get 0 and 0, on every repo, forever. Count it from the heading text and you are counting a string the product edits on its own schedule.
The verdict does not follow the findings list
19 of the 28 final reviews in the first slice reported zero open findings. Three of those still ended with "Needs a closer look". So the heading and the findings list are computed separately, and PR 337469 is the visible case: a summary describing unresolved issues sitting next to a findings field reading None.
The second review in that pair is worth reading closely for another reason. It flagged a "previously missed issue" in code that had not changed since the first review. Same commit, same files, one new finding. Requesting a fresh assessment does not buy you a stable one.
The verdict is usually rendered against a diff that no longer exists
13 of the 28 final reviews in the first slice, and 15 of 27 in the second, sat on a commit that was not the PR head at merge time. The reviewed commit is exposed as commit_id on the review object, and comparing it to head.sha costs one call.
That combination is why the "stale verdict" reading misleads when you attribute it to the reviewer. For human approvals, GitHub records the approved diff state and can dismiss the approval when the diff moves. For a COMMENTED review, nothing is recorded and nothing is refreshed, so an off-head verdict is the normal end state. Report it as a coverage caveat next to any defect count, since the reviewer never saw the final version of the change.
Re-review is rare, and the second review is a delta
In an updated-sorted slice of 28 merged PRs, 24 had exactly one Copilot review. Across the two created-sorted slices, 3 of 55 had more than one. The review bodies invite you to request a fresh assessment, and almost nobody does.
That matters more after the November 6 changelog, which says that on later reviews of the same PR, findings unrelated to your recent changes are collapsed and then hidden. A second review reports the delta against your last push. It does not re-read the whole change. Run a reviewer twice, diff the bodies, and a drop in findings reads like an improvement when it is a narrower scope.
Two reviews, one diff, different output
The same-commit pair is the artifact I would put in front of anyone building a quality score. One diff, one commit, two review requests, two different issue counts, one of them inconsistent with its own findings field.
That is the failure mode of a flaky test: the instrument changes its answer while the input stays fixed. A single-run score from that instrument is a sample of one. The cheap version of the test costs nothing. Open a PR, request a review, request it again on the same commit, compare the two bodies line by line. If the counts move, no quality number you compute from one run survives a re-run.
What to record instead
Five fields, one row per review run:
- The reviewed commit id (
commit_id) and the PR head sha at the time you read the result. - The findings count from the structured part of the body, matched against every shape the body has used, not one string.
- The effort level in use, because Lite, Balanced and Max are different products.
- The verdict heading, stored as text, with the date you read it.
- Whether a second review happened, and what changed between the two.
On effort, the concept docs publish estimated AI credit consumption per review at roughly $0.25 average for Lite, $1.50 for Balanced and $10 for Max, with a warning that consumption rises with PR size and repository custom instructions. The November changelog adds that Balanced now uses three reviewer models with another model combining their findings. A quality number with no effort level attached is not comparable to anything.
On the counter itself, the body format keeps moving. The November 6 update moved the effort label into a compact footer, merged skipped and excluded files into one "Files not reviewed" section, and added explicit attribution fields to the REST and GraphQL APIs so you can identify Copilot's reviews and comments directly. Key your metrics on those fields. A grep for a heading string is a metric that breaks on a UI release.
If you want a gate, buy a gate
Advisory review and required review are separate purchases, and the tooling reflects that. CodeRabbit's configuration reference exposes reviews.auto_approvals.enabled (default false) and states that missing, failed, disabled or stale required assessments withhold approval, which gives you a documented condition set to test against your own PRs. Kodus sits at the other end: self-hosted, AGPLv3, bring your own model, with review rules scoped to an organization, a repository or a path, so the scope and the pass conditions live in your config and you can re-run them. Both are testable in the sense that matters here, which is that you can predict from the docs what happens to the verdict when the diff changes. With Copilot's default configuration the answer is nothing, and the docs say so.
Before comparing tools, settle file scope first. I wrote about the documented skip list and what it does to a head-to-head. And if you are still building the baseline for review time, the fixed-slice protocol is the shape I would reuse.
The number your branch protection sees
Copilot code review produces a comment with a verdict in it. Your branch protection reads a review state, and that state was COMMENTED on all 55 merged PRs I sampled. Count the state, treat the heading as UI text, and record the reviewed commit id next to every result you keep. That is the difference between a metric and a screenshot.
