A model fingerprinting test can return a convincing result while missing an important failure mode:
What if the provider knows when it is being tested?

A provider could recognize a fixed fingerprinting prompt, route that request to the genuine model, and send ordinary production traffic to a cheaper or older model.

The test would pass. The endpoint would still behave differently for real users.

This possibility does not invalidate black-box evaluation. It defines a stronger requirement for the evaluation protocol:
The test should be difficult to recognize, reproducible across controls, and informative when request routing changes.

This question came up while working on the model evaluation workflow at Oken. Oken compares AI model and API behavior through controlled black-box tests. The next challenge is making those tests more robust against provider-side routing decisions and ordinary model drift.
A fixed probe is easy to recognize
A fixed fingerprinting task has several properties that make it vulnerable to pattern matching:

  • the prompt text stays the same;
  • the output length is predictable;
  • the value range is fixed;
  • the response format is unusual;
  • the request structure looks different from ordinary application traffic;
  • the provider can collect and identify the test over time. For example, a provider could detect a known integer-generation prompt and apply a special route: Recognized fingerprint probe ↓ Route to the advertised model ↓ Return behavior consistent with the reference Normal application requests could follow a different route: Ordinary production request ↓ Route to a cheaper or older deployment In that situation, the detector measures the provider’s response to the probe rather than the behavior most users receive in production.

A detector should therefore avoid treating one successful probe as conclusive evidence.
Fresh content preserves the measurement target
The first improvement is to generate fresh probe content for every run.

The surface form can change while the statistical measurement target remains stable. For an integer-sequence task, this could include varying:

  • the exact prompt wording;
  • sequence length;
  • numeric range;
  • formatting instructions;
  • equivalent task descriptions;
  • session order;
  • prompt position. The purpose is not to add randomness for its own sake. The purpose is to prevent the evaluator from relying on one recognizable string.

A useful protocol should preserve the behavioral dimensions being measured:

  • marginal value preferences;
  • positional effects;
  • transition magnitude;
  • repetition behavior;
  • local sequence structure. At the same time, it should change the details that a routing layer could memorize.

The design principle is:
Keep the measurement objective stable while changing the observable form of the probe.

Fresh content does not make a test impossible to target. It raises the cost of recognizing the test and reduces dependence on a single known prompt.
Request shape is another routing signal
Changing the prompt content may not be enough.

Providers can route requests based on metadata and request structure, including:

  • system instructions;
  • tool definitions;
  • streaming mode;
  • response format;
  • context length;
  • requested capabilities;
  • endpoint parameters;
  • application-specific headers. This suggests a paired test.

The same semantic probe can be sent using two request envelopes:
Request A
Minimal evaluation envelope

Request B
Production-shaped envelope with:

  • a normal system prompt;
  • tool schemas;
  • application context;
  • streaming enabled;
  • ordinary request metadata. The probe content remains equivalent, but its external shape resembles a real production request.

If the endpoint behaves differently across the two envelopes, that difference becomes a useful signal for investigation.

A simplified result could look like this:
Same probe content
Minimal envelope: matches reference model A
Production envelope: matches reference model B

Observed divergence: substantial
This does not prove selective routing by itself. The request envelope can change model behavior naturally. A system prompt may alter instruction following, tool definitions may change the model’s context, and streaming may affect generation or post-processing.

The correct interpretation is:
The endpoint shows request-shape-dependent behavioral differences that are consistent with a routing or configuration effect and require further testing.

That wording preserves the signal without turning it into an unsupported accusation.
Same content, two shapes, repeated many times
A single paired comparison is still vulnerable to noise.

A stronger design repeats the same experiment across multiple fresh probes:
Fresh probe 1:
minimal envelope
production-shaped envelope

Fresh probe 2:
minimal envelope
production-shaped envelope

Fresh probe 3:
minimal envelope
production-shaped envelope
The evaluator can then measure:

  • within-envelope variation;
  • between-envelope variation;
  • consistency across repeated pairs;
  • distance from the trusted reference;
  • whether the effect appears only for one prompt variant;
  • whether the effect persists across time. The important quantity is not simply whether two outputs differ. All independent generations differ.

The useful question is whether the difference between request shapes is larger and more stable than the normal variation expected under a trusted control.
A same-day trusted control
Reference-bank distance creates another interpretation problem.

Suppose a tested endpoint becomes more distant from its stored reference. Several explanations are possible:

  • the provider substituted the model;
  • the provider changed the inference configuration;
  • the model itself received an update;
  • the reference bank became stale;
  • the test protocol changed;
  • the endpoint is naturally variable. A distance from an old reference bank cannot separate these explanations on its own.

One practical control is to call a trusted endpoint serving the claimed model on the same day and under the same protocol.

The evaluator can then compare:
Suspect endpoint → current observation
Trusted endpoint → current control
Stored reference → historical baseline
The useful signal becomes differential behavior:
Difference(suspect, trusted control)
rather than only:
Difference(suspect, historical reference)
The trusted endpoint does not need to be perfectly current. It needs to provide a contemporary baseline for the model under the same testing conditions.

This helps distinguish provider-specific divergence from general model drift.

The control itself requires provenance:

  • provider identity;
  • model assertion;
  • test timestamp;
  • request parameters;
  • detector version;
  • reference-bank version;
  • response hashes. Without that information, the control becomes another unverified assumption. Test-retest spread is evidence A detector should report how stable the same endpoint is across repeated runs.

For example, the same endpoint could be evaluated twice within a short interval:
Run 1: closest to Model A
Run 2: closest to Model A
Run 3: closest to Model B
This may indicate:

  • unstable routing;
  • high sampling variance;
  • insufficient probe length;
  • inconsistent provider configuration;
  • an incomplete reference bank;
  • a mixed deployment. The spread itself is informative.

A result that changes dramatically within minutes should be treated differently from a result that remains stable across independent runs.

A useful report should therefore include:

  • repeated candidate rankings;
  • score-margin variation;
  • distance variation;
  • compatibility variation;
  • response-quality variation;
  • time between runs. This is more informative than showing only the strongest result from a single run. The evaluator should distinguish three sources of change A practical monitoring system needs to separate at least three kinds of change. Provider-specific divergence The tested endpoint changes while the trusted same-day control remains stable.

This is more consistent with a provider-side routing or configuration difference.
Shared model drift
Both the tested endpoint and the trusted control move in a similar direction relative to the historical reference.

This is more consistent with a model update or a change shared by both deployments.
Measurement instability
The tested endpoint and the control both show large run-to-run variation under the same protocol.

This suggests that the probe, sampling process, or test configuration may not be stable enough for a strong conclusion.

These categories are not automatically identifiable from one run. They require repeated measurements and controls.
What Oken is designed to help with
Oken’s model evaluation workflow provides a practical way to compare black-box endpoints with reference model behavior.

The workflow can be used to:

  • compare API providers;
  • inspect whether two endpoints with the same model name behave similarly;
  • establish a behavioral baseline;
  • investigate possible model-version drift;
  • repeat tests over time;
  • examine candidate rankings and quality diagnostics. The model fingerprint result should be treated as a behavioral measurement tied to a specific protocol and reference-bank version.

It can help identify an endpoint that deserves further investigation. It does not independently reveal the provider’s internal routing configuration.

You can explore the evaluation workflow at https://www.oken.ai/zh.
A more complete result record
A production-oriented evaluator should preserve enough information to reproduce and audit the result.

For example:
{
"provider_asserted_model": "model-a",
"top_reference_match": "model-a",
"closed_set_probability": 0.94,
"absolute_compatibility": 0.81,
"request_shape": "production_like",
"trusted_control_status": "passed",
"test_retest_spread": 0.07,
"detector_version": "detector-1.2.0",
"reference_bank_version": "bank-2026-11-01",
"calibration_version": "calibration-2026-11-02",
"decision": "behaviorally_consistent"
}
For a suspicious result:
{
"provider_asserted_model": "model-a",
"top_reference_match": "model-b",
"closed_set_probability": 0.88,
"absolute_compatibility": 0.42,
"request_shape": "production_like",
"trusted_control_status": "passed",
"test_retest_spread": 0.31,
"decision": "requires_investigation"
}
The output should preserve the difference between:

  • the model the provider claims to serve;
  • the candidate that ranked first;
  • the absolute compatibility of the sample;
  • the stability of the result;
  • the behavior under different request shapes;
  • the version of the evidence used to reach the decision. What would count as stronger evidence? A responsible evaluation should avoid treating one divergence as proof of selective routing.

Stronger evidence would require a repeated pattern:

  • fresh probes produce the same result;
  • the effect appears across multiple request variants;
  • production-shaped requests behave differently from minimal requests;
  • the trusted control remains stable;
  • the difference exceeds normal test-retest variation;
  • the pattern persists across time;
  • the result is consistent across multiple API keys or sessions;
  • ordinary production traffic shows related behavior. Even then, the conclusion should describe the observation rather than claim access to the provider’s internal routing logic.

A suitable report might say:
Across repeated fresh probes, the endpoint showed a stable request-shape-dependent behavioral difference relative to a same-day trusted control. The pattern is consistent with routing or configuration differences and warrants further investigation.

This is a stronger and more defensible statement than:
The provider is definitely routing test requests to the real model.

Conclusion
A fixed probe can be measured, recognized, and targeted.

A stronger black-box evaluation protocol should use fresh probe content, multiple request shapes, same-day controls, and repeated runs. It should measure test-retest spread and preserve the versions and provenance required to interpret the result.

The objective is not to make the test impossible to game. No black-box test can guarantee that against a provider with complete control of the endpoint.

The objective is to make selective behavior more difficult to hide and easier to detect.

For developers comparing AI API providers, behavioral fingerprinting can provide a useful external signal when deployment logs and routing records are unavailable. Oken provides a practical workflow for running these comparisons and tracking provider-level differences over time.

The result should be read as evidence about observed behavior under a defined protocol. Its strength comes from fresh probes, controls, repeated measurements, and transparent uncertainty.

Disclosure: Oken is the platform discussed in this article. Its evaluation workflow reports behavioral similarity against reference samples and should not be interpreted as cryptographic proof of the server-side model identity.