A generic coding test tells you almost nothing about an AI engineer. A strong backend developer can still get evaluation design wrong. A strong data scientist can still struggle to serve a model under load. The mistakes differ by role, so the interview has to differ too.
These are the design scenarios we use, one per role, with the evidence a candidate has to show to pass. Steal them for your own loop.
Machine learning engineer
Scenario: serve a ranking or forecasting model behind an API with a fixed latency budget, then plan how it gets retrained.
A pass shows:
- an offline metric tied to the business outcome, not just accuracy;
- a test set with no leakage from training;
- a serving plan with its latency and cost stated.
Generative AI engineer
Scenario: question answering over a large set of internal documents, with a ceiling on cost per request.
A pass shows:
- an evaluation set built before any prompt tuning;
- retrieval measured on its own, separate from generation;
- a cost per request they can estimate and defend.
AI agent developer
Scenario: an agent that opens and updates tickets through a ticketing API, where one tool times out halfway through a task.
A pass shows:
- timeouts and retries for each tool;
- actions that are safe to run twice;
- a log of every step;
- human approval where a mistake costs money.
The timeout is the whole point of this one. Most demo agents have never met a tool that fails.
Computer vision engineer
Scenario: a defect detector that has to keep up with a production line camera on an edge device.
A pass shows: a labeling guide with a check on label disagreement, metrics for each class, and latency measured on the device itself, not on a laptop GPU.
NLP engineer
Scenario: pull fields from scanned contracts with inconsistent layouts and OCR errors.
A pass shows: error analysis field by field, a simple baseline before any large model, and a plan for documents the model can't read.
MLOps engineer
Scenario: release a new model version through a canary, with an automatic rollback trigger.
A pass shows: versioned models and data, a rollout that stops itself on a bad metric, drift alerts with an owner, and a rollback they have actually run.
Data engineer
Scenario: design the tables a training pipeline reads from, with late-arriving events and an upstream schema change.
A pass shows: a data model that keeps history, quality checks that stop a bad batch before training, and backfills that can rerun without duplicates.
Two rules that make the scenarios work
- Change a constraint halfway through. Halve the budget or tighten the latency target and watch how the design holds up. Candidates who memorised an architecture stall; candidates who understand it adjust.
- Score each stage on its own. Someone who aces the coding screen and struggles in the design review shouldn't average out into a pass. Write down the evidence behind every score, so the hiring manager can read it later.
We run these as part of a four-stage evaluation at Bromena, and the client still runs its own interviews afterwards. If you use a different scenario for one of these roles, we'd like to hear what it catches that ours misses.
— Bromena team
