Everyone promises AI will train the next generation of clinicians. Here are 7 questions I'd ask first.

Every week there's a new demo of an AI tutor that "thinks like a doctor".
I build learning tools, and I don't think the demo is the hard part.

The demo shows a fluent explanation. Fluency is exactly what language models are good at. What matters for anyone training to make decisions about real people is everything the demo doesn't show. These are the seven questions I'd ask about any AI practice tool for clinical education, and, as you'll see, for coding tutors too.

1. What exactly is being practiced?

"Learning medicine" covers very different skills: recalling facts, recognizing patterns, reasoning under uncertainty, communicating with patients. A tool that's great at flashcard recall may do nothing for reasoning.

Ask: which specific skill does a session make me perform, not just read about?

2. Where does the content come from, and who checked it?

Generated content can be plausible and wrong. In a clinical context, a plausible error is worse than an obvious one, because it gets learned.

Ask: is the material generated live, or drafted and reviewed? By whom? Is there a visible way to report an error, and does anything happen when you do?

3. What happens after a wrong answer?

This is the question I care about most. The research on feedback is clear that its form matters: Hattie and Timperley's The Power of Feedback distinguishes feedback on the task and process from feedback on the person, and only the first kinds reliably help.

Ask: after I'm wrong, does the tool tell me why my reasoning went wrong, or just show the right answer?

4. Does it make me answer before it explains?

Retrieval practice, answering from memory before seeing the answer, beats restudying for long-term retention (Roediger & Karpicke, 2006). A chatbot that explains on request makes rereading effortless, which is the opposite.

Ask: does the default flow force me to commit to an answer first?

5. How does it handle uncertainty?

Real clinical reasoning is full of "it depends" and "we need more information". Language models tend to sound certain either way.

Ask: does the tool ever say "this is uncertain" or "the evidence is mixed"? Does it cite sources I can check? Does it model when to ask for more data?

6. Does it build a habit, or a binge?

Deliberate practice depends on repetition over time (Ericsson et al., 1993). A tool that encourages a five-hour session before an exam is a cramming tool. One that brings you back to small, finishable sessions is a practice tool.

Ask: is a session small enough to do on a terrible day? Does it bring back what I got wrong last week?

7. How is it evaluated?

"Users love it" and "engagement is up" are not learning outcomes. Neither is "the model passes a licensing-style exam", which says something about the model, not about the learner.

Ask: is there any evidence that people who use it get better at the skill, measured independently? If not, is the team at least honest that it's unknown?

The same seven questions for coding tutors

Swap "clinician" for "junior developer" and almost nothing changes:

1. Does it make me write code, or read code?
2. Who checked the examples?
3. After a failing test, does it explain my mistake?
4. Do I attempt before it suggests?
5. Does it admit when an approach is debatable?
6. Does it help me practice a little every day?
7. Do people actually get better, or just faster at copying?

The stakes differ. The design problem doesn't. In both fields, the tool that feels most helpful in the moment, the one that just gives you the answer, can be the one that teaches the least.

What I'm not saying

I'm not saying AI can't help clinicians learn. Generated variations of practice material, explanations after an attempt, and spaced review are genuinely useful. I'm saying the value lives in the feedback loop design, and the demo rarely shows it.

A quick scorecard

If you're evaluating a tool for yourself, your study group or your team, give it one point per "yes":

Seven out of seven is rare, and that's fine. What matters is knowing which ones are missing, so you can fill the gap yourself: a study partner for feedback, a textbook for checked content, a calendar reminder for the habit.

A tool that scores two out of seven can still be useful, as long as you don't mistake it for the whole training.

Takeaways

1. Fluency is cheap for language models. Judge tools on the loop, not the explanation.
2. Know which skill a session makes you perform.
3. Reviewed content beats live-generated content where errors are costly.
4. Answer first, explanation after.
5. Ask for evidence of learning, not engagement.

Your turn

If you're a clinician, a student, or a developer who's used an AI tutor seriously: which of these seven questions did your tool fail, and which would you add?

I write about practice, feedback and the AI-readable web while building NEXO, a daily learning game.