The study sits on the 1,503-row Airfoil Self-Noise set from Brooks, Pope, and Marcolini (1989), hosted on UCI as dataset 291 (DOI 10.24432/C5VW2C). It has five inputs: frequency in Hz, attack angle in degrees, chord in meters, free-stream velocity in m/s, and suction-side displacement thickness in meters, plus one target, a scaled sound-pressure level in dB. It has no experiment ID. UCI's variable table labels the angle column "Binary," while the file itself holds 27 numeric angles from 0 to 22.2; I kept the real values.
The missing ID is where the problem starts. Holding a test row out of training only holds that exact row out. Another row from the same physical setup, at a nearby frequency, can stay in the training set. A model that has seen that neighbor answers a different question from one that has not, even when both are scored on the same rows.
The condition proxy and the two pools
I built a condition proxy from the four non-frequency features: angle, chord, velocity, and displacement thickness. Canonicalizing each value and joining them gives 106 exact tuples across the 1,503 rows. These tuples are not run IDs. Two rows sharing a tuple may still depend on each other through factors the dataset does not record.
From those tuples I split the data before any model existed, using a hash of each tuple's canonical string. SHA256 of the tuple, taken modulo 10, assigns each row to a group: remainders 0 and 1 become test proxies (27 groups), 2 and 3 become validation proxies (20 groups), and 4 through 9 become base fit groups (59 groups). Inside each test group I sort rows by numeric frequency, then by original row order, and take even ranks as the common test set (201 rows) and odd ranks as context rows (185 rows). Validation uses all 274 rows of its groups. Membership is decided by hashes of feature tuples, never by target values. That is a label-blind split, but the study is not historically blind: I had already read all feature and target values before writing the protocol.
Two training pools share the same 843-row budget and the same 274 validation and 201 test rows.
- Pool A (condition_disjoint) uses all 843 base rows. It never sees any of the 27 test condition proxies.
- Pool B (row_context) uses all 185 context rows plus 658 base rows. The base rows are ranked lexicographically by SHA256 of "row-context-replace:" plus the full five-feature key, ties broken by original row order, and the first 658 are kept. B therefore replaces 185 of A's rows with the context rows.
Pool B sees other frequencies of all 27 test proxies; pool A sees none. No identical test row enters either fit. That is the one deliberate difference in the question, but the replacement also shifts which rows are covered and how each pool's normalizers are fit.
Six small networks
Each network is 5 → 24 tanh → 1 linear, 169 parameters. I trained two pools times three fixed seeds (17, 41, 73), six fits in total, each for exactly 120 epochs with batch size 32, Adam at learning rate 0.003, and a weight-only L2 of 0.0001, in NumPy float64 on CPU. For a given seed the initial parameter arrays are identical across the two pools, but each pool normalizes its inputs and target with its own 843 rows, so the initial function is not necessarily the same after normalization. Input means and population standard deviations, target mean and standard deviation, and the constant baseline all use only that pool's training rows.
Each fit runs 3,240 updates and 101,160 row visits; all six together run 19,440 updates and 606,960 row visits. There is no early stopping and no sweep. The selected epoch for each model is the earliest minimum of common-validation MAE in dB; test rows never participate in selection. Selected epochs are A 116/110/118 and B 119/117/116. The six fits were completed on November 4, 2026, and this release reuses those frozen models and results, adding no new training.
The numbers
The table shows test MAE and RMSE for all six networks on the same 201 rows, in the target's dB units.
| Pool | Seed | Test MAE (dB) | Test RMSE (dB) |
|---|---|---|---|
| A | 17 | 2.637440 | 3.433836 |
| A | 41 | 2.685900 | 3.475715 |
| A | 73 | 2.610386 | 3.387189 |
| B | 17 | 2.268305 | 2.979579 |
| B | 41 | 2.301926 | 2.995912 |
| B | 73 | 2.399225 | 3.197637 |
The arithmetic mean of the three seed MAEs is 2.644575 dB for A and 2.323152 dB for B. That is a mean of three already-computed metrics, not an ensemble prediction and not a confidence interval.
The mean does not cover every row. Pairing the two pools row by row on the same 201 test cases, once per seed, gives 603 paired comparisons. Those reuse the same 201 rows three times and are not 603 independent examples. In 237 of them, B has a larger absolute error.
| Seed | B smaller | B equal | B larger |
|---|---|---|---|
| 17 | 115 | 0 | 86 |
| 41 | 135 | 0 | 66 |
| 73 | 116 | 0 | 85 |
The default case shown by the page is one of those. It is the first common-test source row (row 29) with seed 17, fixed before any model score was seen. Its recorded target is 118.129 dB. A predicts 124.342 dB (absolute error 6.213 dB) and B predicts 125.166 dB (absolute error 7.037 dB), so B is worse on the first case a visitor sees. I kept that case rather than substituting a winning seed.
Two train-mean baselines, one per pool (a single analytic mean each, not six more fits), score test MAE 5.853341 dB and 5.871245 dB. Those are numerically higher than every one of the six networks on this fixed test set, which tells me the networks learned something beyond guessing the pool mean. I did not run a significance test on any of these comparisons.
What the comparison supports and what it does not
The supported claim is narrow. With the same budget, the same validation and test rows, the same architecture and optimizer, and paired seeds, the two pools produce different mean test MAEs, and B's is lower. That difference does not cover every case: 237 of 603 paired rows go the other way.
I will not claim that seeing the test condition caused the improvement. Replacing 185 rows changes training coverage and the fitted normalizers at the same time it changes condition exposure. Each pool's input and target statistics are computed from its own 843 rows, and those differ. I cannot pull those apart here and assign the gap to overlap alone.
Pool B also does not estimate error on unseen physical conditions. Its numbers describe a task where the test conditions' other frequencies were already in training. Pool A is a local control where the test condition proxies never entered training, but the proxy is a tuple, not a physical experiment ID, and unrecorded dependencies may remain between different tuples. None of this establishes physical experiment independence, noise compliance, or any engineering guarantee.
Metric details
The primary test MAE uses all 201 rows, a micro average. results.json also stores a macro value that averages the 27 test proxy group MAEs equally; both are in the file and neither replaces the other. The three-seed mean averages three already-computed metrics; I did not average three outputs and then measure error, and it is not a confidence band. The target unit is scaled sound-pressure dB, not A-weighted dBA, and not a pressure or energy quantity. The 603 paired rows reuse the same 201 test cases three times, so I report per-seed counts and descriptive means, treating 603 as a denominator of paired predictions rather than independent samples for a test.
Recomputing the numbers
First get the full files at the fixed commit 8ff0d17f86ce414c7eb4fba347024d443845a80e, using the Files view rather than just the running App. Once they are local, run the block below from the package root, the directory that contains evaluation-001/. The script itself makes no network request, trains nothing, writes nothing, and uses only the Python standard library. It reads evaluation-001/results.json and evaluation-001/paired-test.csv.
python3 - <<'PY' import json, csv with open("evaluation-001/results.json", encoding="utf-8") as f: r = json.load(f) neural = [m for m in r["overall"] if m["role"] == "test" and m["kind"] == "neural"] for m in sorted(neural, key=lambda x: (x["policy"], x["seed"])): print(f"{m['model_id']:22s} test MAE = {m['mae_db']:.6f} dB") print() for policy in ("condition_disjoint", "row_context"): vals = [m["mae_db"] for m in neural if m["policy"] == policy] print(f"{policy:22s} mean of 3 seed MAEs = {sum(vals)/len(vals):.6f} dB") print() with open("evaluation-001/paired-test.csv", encoding="utf-8", newline="") as f: rows = list(csv.DictReader(f)) for seed in (17, 41, 73): sel = [x for x in rows if int(x["seed"]) == seed] worse = [x for x in sel if float(x["row_context_minus_disjoint_absolute_error_db"]) > 0] d = [float(x["row_context_minus_disjoint_absolute_error_db"]) for x in sel] print(f"seed {seed}: paired {len(sel)}, B worse {len(worse)}, " f"mean delta {sum(d)/len(d):.6f} dB") total_worse = sum(1 for x in rows if float(x["row_context_minus_disjoint_absolute_error_db"]) > 0) print(f"\ntotal paired {len(rows)}, total B-worse {total_worse}") PY The block prints the six test MAEs, the two three-seed means, the per-seed paired counts and mean deltas, and the total B-worse count. Recomputing saved predictions is verification, not another fit.
The local page
The running space is WideChaos/airfoil-split-lab, and the static App is here. The App runs the six fixed models in the browser with no model API and no server-side storage. Inputs live only in the page, so a refresh loses edits. The exported report keeps all six outputs, the full input values, the normalizers, and the hashes. Those hashes bind consistency, not authenticity.
Four actually downloaded native reports keep all 24 outputs. A separate scalar reimplementation compares them with a largest difference of about 1.42e-14 dB. A different portable check over all 475 validation/test contexts times six original models agrees within 2.843e-14 dB. These are consistency checks on the product, not a new experiment and not a claim about privacy or reliability on arbitrary new inputs.
The figure
The four panels show all six 120-epoch validation curves with their selected epochs, the six test MAEs on the same 201 rows, the 603 paired error differences (237 positive, meaning B has the larger absolute error), and training proxy exposure (A sees 0 of the 27 test proxies, B sees all 27). It is a rendering of the frozen results, not a screenshot of the page and not a causal diagram.
Source and license
The protocol is in PROTOCOL.md at the fixed commit. The numbers live in evaluation-001/results.json and evaluation-001/paired-test.csv, both linked above. Code and documentation are ©2026 Renchao Wu, Apache-2.0. The original data and the derived weights, normalizers, results, and figures carry Brooks/Pope/Marcolini/UCI CC BY 4.0 attribution; the notices are in NOTICE and LICENSES.md. NASA-RP-1218 (record) is cited as a source record only; I read its record, abstract, and metadata, and nothing from that report is copied or relicensed. UCI's variable table labels the angle column "Binary," but the file holds 27 numeric angles, and I kept those real values rather than binarizing them. That is a record of the discrepancy, not a change to the upstream metadata.
