A three-step prompt chain — plan, draft, review — works beautifully in a notebook and falls apart the first week it runs unattended. Not because the model is bad, but because nothing in the chain checks what each step hands to the next one, and nothing decides how much money a failure is allowed to spend before it starts.

This post is the small amount of structure that fixed that for me: three steps, a contract per step, two different recovery strategies, and a cost ceiling that is enforced before a request leaves the process. Everything is standard library. No SDK, no framework, no dependency to pin.

The four failure modes I kept hitting:

  1. Type drift. Step 1 returns "3", step 2 was written for 3, and step 3 silently renders nothing.
  2. Wrapped output. The model answers with a friendly sentence, then a fenced block, then JSON — and occasionally a trailing "Let me know if you want me to expand this!"
  3. Cost creep. A retry loop that retries a validation failure at the most expensive model in the list, four times, at 3 a.m.
  4. One provider, no plan. The first 429 ends the run, even though a free tier that is nobody's bottleneck was already configured.

Step 1: Give every hop a contract

A prompt is not a schema, and "please return JSON" is not validation. I describe each step's output with plain dataclasses — one Field per key — and validate the parsed reply against it. That is enough to catch everything on the list above, and it is ~50 lines you can read in one sitting.

from dataclasses import dataclass, field from typing import Any, Mapping, Sequence import json, time class ValidationError(ValueError): """The model answered, but the answer is not what the step promised.""" @dataclass(frozen=True) class Field: name: str kind: type min_len: int = 0 max_len: int | None = None choices: tuple[str, ...] | None = None minimum: int | None = None maximum: int | None = None item: "Field | None" = None # list of scalars of: tuple["Field", ...] | None = None # list of objects def opts(self) -> dict: out = {} if self.min_len: out["min_len"] = self.min_len if self.max_len is not None: out["max_len"] = self.max_len if self.choices is not None: out["choices"] = self.choices if self.minimum is not None: out["minimum"] = self.minimum if self.maximum is not None: out["maximum"] = self.maximum return out def _check(obj: Mapping[str, Any], f: Field) -> Any: if f.name not in obj: raise ValidationError(f"missing field {f.name!r}") v = obj[f.name] if f.kind is int: # bool is an int subclass in Python; reject it on purpose. if isinstance(v, bool) or not isinstance(v, int): raise ValidationError(f"{f.name!r} must be int, got {type(v).__name__}") if f.minimum is not None and v < f.minimum: raise ValidationError(f"{f.name!r} is below {f.minimum}") if f.maximum is not None and v > f.maximum: raise ValidationError(f"{f.name!r} is above {f.maximum}") return v if f.kind is list: if not isinstance(v, list): raise ValidationError(f"{f.name!r} must be a list") if f.max_len is not None and len(v) > f.max_len: raise ValidationError(f"{f.name!r} has more than {f.max_len} items") if f.item is not None: rows = [] for i, it in enumerate(v): key = f"{f.name}[{i}]" rows.append(_check({key: it}, Field(key, f.item.kind, **f.item.opts()))) return rows if f.of is not None: rows = [] for i, it in enumerate(v): if not isinstance(it, dict): raise ValidationError(f"{f.name}[{i}] must be an object") rows.append({sub.name: _check(it, sub) for sub in f.of}) return rows return v if not isinstance(v, f.kind): raise ValidationError(f"{f.name!r} must be {f.kind.__name__}, got {type(v).__name__}") if isinstance(v, str): v = v.strip() if len(v) < f.min_len: raise ValidationError(f"{f.name!r} is shorter than {f.min_len} chars") if f.max_len is not None and len(v) > f.max_len: raise ValidationError(f"{f.name!r} is longer than {f.max_len} chars") if f.choices is not None and v not in f.choices: raise ValidationError(f"{f.name!r} must be one of {list(f.choices)}, got {v!r}") return v 

Three details earn their keep. bool is rejected where an int is required, because True will happily pass an isinstance(x, int) check and then break an arithmetic step three hops later. choices turns free text into an enum, which is the single cheapest guardrail against a model inventing "looks good" where you asked for "ship". And every error message names the field and the rule, because that message becomes the repair prompt in a moment.

A step is a name, a system prompt, its fields, and — this matters for cost — an explicit list of which upstream keys it is allowed to see.

@dataclass class Step: name: str system: str fields: tuple[Field, ...] = () needs: tuple[str, ...] = () # upstream keys it sees max_repairs: int = 1 clip: dict[str, int] = field(default_factory=dict) # "key.subkey" -> chars def validate(self, payload: Any) -> dict[str, Any]: if not isinstance(payload, dict): raise ValidationError("top-level JSON must be an object") return {f.name: _check(payload, f) for f in self.fields} def shape(self) -> dict[str, Any]: out = {} for f in self.fields: if f.of is not None: out[f.name] = [{s.name: s.kind.__name__ for s in f.of}] elif f.item is not None: out[f.name] = [f.item.kind.__name__] else: out[f.name] = f.kind.__name__ return out 

shape() also solves failure mode 2 at the source: the prompt shows the model the exact keys and Python types it must return, serialized with json.dumps(..., separators=(",", ":")) and no prose around it.

Step 2: Parse defensively, because the fence is coming

Even with a shape in the prompt, replies arrive fenced, prefixed, or suffixed. Find the outermost braces and parse only that span — and turn a parse failure into a ValidationError so it takes the same repair path as a wrong type.

FENCE = "`" * 3 def _load_json(raw: str) -> Any: """Models love wrapping JSON in prose or fences. Strip that before parsing.""" t = raw.strip() if t.startswith(FENCE): t = t.split("\n", 1)[-1] t = t.rsplit(FENCE, 1)[0] start, end = t.find("{"), t.rfind("}") if start == -1 or end <= start: raise ValidationError("no JSON object found in the reply") try: return json.loads(t[start:end + 1]) except json.JSONDecodeError as e: raise ValidationError(f"invalid JSON: {e.msg} at char {e.pos}") from e 

If you have ever written a regex to strip code fences, note that the version above handles a fence with a language tag and a closing fence on its own line, and nothing more — which is exactly as much cleverness as this deserves. FENCE is built as "`" * 3 so that this very article's code block does not contain the characters that would close it. Leftover edge cases land in the repair path, where they belong.

Step 3: Two recovery strategies, not one

This is the part I got wrong for a long time. A transport failure (429, timeout, 5xx) and a contract failure (valid HTTP, wrong shape) are not the same event, and retrying them identically is how you burn an evening of quota.

  • Transport failure → same step, same prompt, short backoff, then the next provider down the ladder.
  • Contract failure → same step, different prompt: the complaint, and nothing else. Re-sending two kilobytes of broken output to tell the model it was broken is the most expensive way to say "no".
@dataclass(frozen=True) class Provider: name: str call: Callable[[str, str], str] # (system, prompt) -> raw text tier: int = 0 # 0 free/local, 1 free tier, 2 paid class BudgetExceeded(RuntimeError): pass class ChainRunner: def __init__(self, providers: Sequence[Provider], *, max_tier: int = 0, max_prompt_tokens: int = 1500, sleep: Callable[[float], None] = time.sleep, backoff: float = 0.0) -> None: self.providers = list(providers) self.max_tier = max_tier self.max_prompt_tokens = max_prompt_tokens self.sleep = sleep self.backoff = backoff self.spent_tokens = 0 @staticmethod def estimate_tokens(text: str) -> int: return max(1, len(text) // 4) # rough but stable for English/JSON def _guard(self, system: str, prompt: str) -> None: n = self.estimate_tokens(system) + self.estimate_tokens(prompt) if n > self.max_prompt_tokens: raise BudgetExceeded(f"prompt is ~{n} tokens, over the {self.max_prompt_tokens} cap") def _ask(self, step: Step, prompt: str) -> tuple[str, str]: last = None for p in [p for p in self.providers if p.tier <= self.max_tier]: for attempt in range(2): # one try, one retry, that is all try: return p.call(step.system, prompt), p.name except Exception as e: # 429 / 5xx / socket last = e if attempt == 0 and self.backoff: self.sleep(self.backoff) raise RuntimeError(f"every provider at tier <= {self.max_tier} failed: {last!r}") def run(self, steps: Sequence[Step], seed: Mapping[str, Any]) -> dict[str, Any]: state = dict(seed) trace = [] for step in steps: prompt = step.prompt(state) self._guard(step.system, prompt) # before the network, not after repairs = 0 while True: raw, provider = self._ask(step, prompt) self.spent_tokens += self.estimate_tokens(prompt) + self.estimate_tokens(raw) try: out = step.validate(_load_json(raw)) except ValidationError as e: trace.append({"step": step.name, "provider": provider, "ok": False, "error": str(e)}) if repairs >= step.max_repairs: raise repairs += 1 # send back the complaint, not the broken output prompt = step.prompt(state, note=str(e)) continue trace.append({"step": step.name, "provider": provider, "ok": True, "repairs": repairs}) state[step.name] = out break state["_trace"] = trace return state 

max_tier is the cost policy, and it is a filter on the provider list rather than a flag inside each provider — so a provider that costs money is not merely discouraged, it is unreachable when the ceiling is 0. _guard measures the prompt before _ask is called; a runaway context is a free, local, instant exception instead of a bill. And max_repairs=1 per step is deliberate: if the model cannot satisfy the contract in two tries, the prompt or the contract is wrong, and a third try is just a more expensive way to learn the same thing.

Putting the chain together

outline → draft → review, where review deliberately does not receive the whole draft.

OUTLINE = Step( name="outline", system="You are a technical editor. Plan short, concrete developer posts.", needs=("topic", "audience"), fields=( Field("promise", str, min_len=20, max_len=160), Field("sections", list, max_len=6, of=( Field("heading", str, min_len=3, max_len=80), Field("intent", str, min_len=10, max_len=200), )), ), ) DRAFT = Step( name="draft", system="You are a senior engineer writing runnable, copy-pasteable code.", needs=("promise", "sections"), fields=( Field("title", str, min_len=15, max_len=100), Field("body_markdown", str, min_len=400), ), ) REVIEW = Step( name="review", system="You are a strict reviewer. Reject what you cannot verify.", needs=("promise", "draft"), clip={"draft.body_markdown": 1800}, # head + tail, not the whole body fields=( Field("score", int, minimum=0, maximum=100), Field("verdict", str, choices=("ship", "revise")), Field("issues", list, max_len=5, item=Field("issue", str, min_len=5, max_len=Source:https://dev.to/yuya_ai_lab/how-to-build-a-reliable-3-step-chained-ai-prompt-pipeline-in-python-with-schema-validation-and-3lc3

Related posts