An aggregate pass rate compresses a dataset into a convenient headline. Before using it, inspect the room-level counts that feed it. In the independent StagingBench test, every room has results from 12 tools, but the balance of pass, fail, and unknown differs substantially between rooms.
The dataset is a small test of 10 real room photographs, with one recorded run per tool and room cell. It cannot establish repeatability. It can support a narrower audit: checking how much of each room's result set has a definite framing outcome before interpreting a summary.
Count outcomes before calculating a rate
For this check, a row with alignment_ok set to False is unknown. For other rows, framing_pass determines whether the row counts as pass or fail. Checking unknown status first prevents a fallback row from being counted as a definite outcome.
This snippet uses only Python's standard library. Run it from the directory containing scores.csv from the open dataset:
import csv from collections import Counter, defaultdict counts = defaultdict(Counter) with open('scores.csv', newline='') as handle: for row in csv.DictReader(handle): if row['alignment_ok'] == 'False': status = 'unknown' else: status = 'pass' if row['framing_pass'] == 'True' else 'fail' counts[row['room']][status] += 1 for room, c in sorted(counts.items()): print(f"{room}: pass={c['pass']} fail={c['fail']} " f"unknown={c['unknown']}") The output is a room audit, not an ordering of tools. It also keeps unknowns visible instead of silently dropping them from the denominator.
Read the rooms with many unknowns first
Room 09 has zero passes, zero failures, and 12 unknowns. That means its recorded results provide no definite framing outcomes. Reading the missing passes as evidence that every tool failed would change the meaning of unknown.
Room 04 has three passes, one failure, and eight unknowns. A pass rate stated as three out of 12 describes passes among all recorded cells. A statement of three passes among the known outcomes uses a different denominator. Either requires an explicit description of what is being counted.
Room 06 has five passes, five failures, and two unknowns. Rooms 02 and 03 each have 10 passes, one failure, and one unknown. Rooms 07 and 10 each have nine passes, two failures, and one unknown.
These counts identify where uncertainty sits. They do not identify why it occurs, and they say nothing about the visual features of those rooms.
Also inspect rooms without unknowns
Room 01 has eight passes, four failures, and zero unknowns. Room 05 has 10 passes, two failures, and zero unknowns. Room 08 has five passes, seven failures, and zero unknowns.
Together with the other room counts, these results show why a shared number of submissions does not guarantee a shared amount of interpretable evidence. Each room contributes 12 recorded cells. Some contribute definite outcomes throughout; others contribute a mixture of definite outcomes and unknowns.
Absence of unknowns is not proof of overall image quality. This audit concerns the fields used for the framing count. The StagingBench methodology describes the measurement context. A definite framing result should not be expanded into a claim about every aspect of a staged image.
State the denominator with the claim
Before publishing a pass rate, decide whether the denominator includes all recorded cells or only rows with definite outcomes. State that choice alongside the result. Keep the per-room counts available so readers can see what the aggregate compresses.
For example, room 09 needs an explicit unknown count whichever summary convention is chosen. Excluding its unknown rows does not create evidence about that room. Including them in a denominator does not turn them into observed failures.
This room-level check is useful precisely because it is modest. It does not explain the outputs, diagnose the tools, or establish a general success rate for virtual staging. It shows which observations support a definite count and which remain unknown in this particular run. That is a necessary context for deciding how much weight an aggregate deserves.