The model proposes, the code decides: automatic diagnosis of dbt failures with draft pull requests

By Ricardo Ribeiro · Open source (Apache-2.0): github.com/ricardoribs/pipeline-guard

It is 3 a.m. and a dbt build fails. Someone wakes up, opens the log, reads 400 lines and discovers that the problem was a single line of SQL. The fix takes two minutes. Finding the cause took an hour.

That hour is what I wanted back. I built Pipeline Guard, an open-source tool that reads the failure, diagnoses the cause using evidence you can check, fixes it in an isolated environment, tests the fix and opens a draft pull request. A person reviews it and decides.

In this article I explain the idea behind it, list everything it delivers, show how each step works and present two real test cases in a sandbox.

The problem

A data pipeline is a sequence of steps that turns raw data into tables ready for analysis. dbt is the tool that handles a large part of those transformations, written in SQL. When a step fails, the team has to find out what broke, where and why.

The obvious reaction is to let a language model fix everything on its own. The risk is big: the model can get the cause wrong, invent a convincing explanation, edit a file it should not touch, or expose data and credentials along the way. In data work, a silent mistake costs more than a loud failure.

The question that guided the project was this: how do you use the model without handing control over to it?

The idea: separate who reasons from who decides

The answer was to split the work into two sides.

• The model reasons. It reads the evidence, classifies the failure and proposes a fix.
• Deterministic code decides. Deterministic means that for the same input the answer is always the same, with no improvisation. That code validates every citation the model makes, applies the eligibility rules, checks the authorization and controls everything that can happen in the real world.

In one sentence: the model proposes, the code decides.

The model, for example, has no operation to approve a PR, merge, deploy or rerun the pipeline that failed. That does not depend on it behaving well: that path simply does not exist in the code.

What Pipeline Guard delivers

For each failure, it delivers a complete, verifiable package, not just a chat answer. These are the ten deliverables, in the order they happen:

1. Validation of the incident and the configuration. Before collecting any evidence, it checks that the incident and the project configuration file follow the contract. If something is invalid, the run stops with its own exit code.
2. Read-only evidence collection. An adapter from a closed list (today dbt and mock) collects only what the incident declared. In dbt, those are the run_results.json and manifest.json files, with hash checks.
3. Sanitization before the model. Sensitive fields, authentication headers, cookies, private keys, email addresses, signed URLs and known token formats are masked before anything is sent to the model or written to disk.
4. A structured diagnosis with citations. The result follows a versioned contract (pipeline_diagnosis_v1): cause, failure family, confidence, cause status, impact and the evidence cited. If the evidence is insufficient or conflicting, the cause becomes UNKNOWN instead of a guess.
5. An eligibility decision. Deterministic code, not the model, says whether the failure can move on to a fix.
6. A neutral report. Output in JSON (for machines) and in Markdown (for people), without depending on Slack, Airflow, Git or a warehouse.
7. Immutable artifacts and a manifest. Each run stores the result, diagnosis, decision and report, plus a manifest with the final status, contract versions, evidence hashes, timestamps and duration. Repeating the same incident reuses the final result instead of creating a conflicting duplicate.
8. A minimal fix in an isolated workspace. With verified authorization, a remediation agent edits only what is needed, within limits on path, file extension, number of files and number of lines.
9. Validation with your project's own commands. It runs only the commands the project declared (in dbt, deps and build) and checks the patch hash.
10. A restricted push and a draft PR. It pushes only the incident branch, with a strict refspec, revalidates the pushed commit and creates or reuses a draft PR on Bitbucket Cloud or GitHub. The description carries the incident, the diagnosis, the evidence, the changed files and the test results.

The final result is versioned and machine-readable, and comes with stable exit codes that any orchestrator knows how to read:

Code 3 is a controlled result. The tool prefers to stop rather than infer a cause it cannot prove.

How it works, step by step

1. Normalized incident. The failure enters in a standard format: which pipeline, which step, which environment, which commit.
2. Traceable evidence. The dbt adapter reads the run_results.json and manifest.json files that dbt itself generates and checks that their hashes (each file's fingerprint) match what the incident declared. Each piece of evidence gets an identifier and is sanitized before the diagnosis.
3. Diagnosis without tools. The agent only classifies the failure. It executes nothing. Every citation it makes is validated against the evidence collected.
4. Eligibility. A fixed rule decides whether the case can move on to a fix. Today, only failures of type CODE (an error in the code), with verified authorization tied to the incident, the diagnosis and the base commit.
5. Fix in an isolated area. A separate workspace, editing only the paths the project configuration allows and running only the validation commands the project declared.
6. Draft PR. If everything passed, the tool creates (or reuses) a draft pull request. A person approves and merges.

Two ways to use it

The same core serves someone investigating a failure alone and someone who wants the repair running inside their orchestrator.

Interactive, in Claude Code

Pipeline Guard ships as a plugin with four skills (sets of instructions that Claude activates depending on what you ask):

• pipeline-guard-setup: installs and configures the plugin in a project, generating and validating the configuration file from a preset.
• diagnose: investigates a failure and returns a validated diagnosis, an eligibility decision and a neutral report.
• remediate: applies an authorized fix, validates the patch and pushes the incident branch.
• publish-draft-pr: publishes or reuses the draft PR after the branch has been pushed.

You ask in plain language, for example: Diagnose the incident described in ./incidents/failed-run.json. The request passes only the file path, never the log pasted into the prompt. A fail-closed hook blocks diagnosis prompts that contain recognized sensitive data before Claude processes them.

Unattended, in any orchestrator

To run at night, with nobody to answer questions, there is a single, host-neutral command:

uv run scripts/unattended_cli.py \ --mode unattended \ --config ./config/pipeline-guard.yaml \ --project-root . \ --output-dir /retained/pipeline-guard/output \ --authorization-state-dir /retained/pipeline-guard/authorization-state \ ./incident.json 

In this mode:

• nothing is asked or confirmed: authorization comes from a versioned project policy and a service identity;
• the work happens in an isolated, managed workspace, which is kept clean when publication has to be safely resumed;
• the output is a versioned, machine-readable result, with a summary for people and stable exit codes;
• the draft PR goes out through Pipeline Guard's own publication boundary, without depending on another plugin;
• there are guides for Databricks Jobs, Bitbucket Pipelines and Airflow, with no host-specific logic embedded in the code;
• a Dockerfile prepares container execution, with an unprivileged user and pinned dependency versions.

Any model, configuration per project

Pipeline Guard does not lock you into one vendor or one kind of project.

• Six model profiles: anthropic (the default), databricks, openai, gemini, ollama and openai-compatible. That includes running with a local model, without sending anything to the cloud. For the agent that writes the fix, the endpoint has to offer function calls in the Chat Completions format.
• A clear order of choice: first the command-line arguments, then the runtime.model block in the configuration file, then environment variables.
• Credentials by reference: the file points to the environment variable and never stores the key as text.

runtime: model: provider: ollama name: llama3.3 base_url: http://127.0.0.1:11434/v1 

• Project presets: generic and dbt. The setup skill generates config/pipeline-guard.yaml from one of them and validates the result.
• Fix policy in your file: you define which paths and extensions can be edited, how many files and lines at most, and which validation commands run.
• Adapters with a single contract: every adapter follows the same contract (pipeline_adapter_interface), the registry is closed and each tool's details stay isolated under adapter_details.<name>. Today there are dbt and mock (the latter to validate an installation without depending on anything external).
• Two PR platforms: Bitbucket Cloud and GitHub.

The safeguards

• Diagnosis without tools. The agent that diagnoses does not read files, run commands, access the network or change the project. It only sees the validated incident and the evidence collected.
• Every claim cites evidence. Each relevant point in the diagnosis must point to an evidence identifier that exists, and an external validator checks the contract and the citations.
• Eligibility by code. Whether the failure can move on to a fix is decided by deterministic code, never by the model.
• Tied authorization. It is valid for one incident, one diagnosis hash and one base commit. If any of the three changes, it stops being valid.
• A contained remediation agent. It receives only the context it needs, uses restricted read, write and edit tools, and has no access to Git or provider credentials. It does not publish, merge or deploy. The step that writes the fix refuses to run with administrator privileges.
• Redaction of sensitive data. Before reaching the model or being stored, text goes through a filter that masks sensitive fields, authentication headers, cookies, session identifiers, private keys, email addresses, signed URL parameters, Azure access signatures and known token formats (such as those from Atlassian and DigitalOcean). Labels are recognized in both Portuguese and English.
• Credentials only through environment variables. They never appear in a prompt, a command argument, a log or a result.
• An execution budget. There are limits on tool calls, time and tokens. A run that is invalid, has insufficient or conflicting evidence, or is out of budget stops safely.
• Writes only where they should happen. Diagnosis writes nothing to the project, Git, the warehouse or a deployment. The fix writes only inside the isolated workspace, pushes only the authorized branch and publishes only a draft, after revalidating everything.
• A narrow publisher. The component that creates the PR can only locate, create, reuse or close the draft that Pipeline Guard itself produced. Approving, reviewing, merging or administering the repository: none of that exists in it.

Two real test cases

The two cases below ran in a test environment (sandbox) on Databricks Jobs, from reference scenarios: failures planted on purpose to exercise the full path. They are not production incidents. The PRs were published to a Bitbucket repository.

Case 1: the inverted test

• Failure: dbt build finished with exit code 1 in the dbt_build step.
• Diagnosis: a test failed because it returned one result when it was configured to expect zero. Failure family: CODE. Confidence: high. Cause: confirmed.
• Evidence: the dbt evidence record reported one failing record.
• Fix: one file changed, tests/scenario_singular_inverted.sql, with 1 line added and 1 line removed.
• Validation: deps passed in 3.9 s and build passed in 5.1 s. Status: all passed.
• Result: PR #20 was created as a draft by Pipeline Guard's publisher. Both of the repository's own pipelines passed. I reviewed it and merged it into the dev branch.

Case 2: the column that went missing

• Failure: the same kind of error, in the same step, but for a different reason.
• Diagnosis: the transformation used a field that was not available at execution time. Family: CODE. Confidence: high. Cause: confirmed.
• Evidence: the execution evidence reported that a referenced field was unavailable.
• Fix: one file, models/scenarios/scenario_missing_column.sql, with 1 line added and 1 line removed.
• Validation: deps in 4.3 s and build in 5.2 s, all passed.
• Result: PR #21, also created as a draft, reviewed and merged by me.

One detail I like: the base commit of the second incident is exactly the merge commit of the first. The cycle works as a chain: the approved fix from one case becomes the base of the next.

What the two cases show

Both cases went through the complete path, end to end: from the failure to a reviewable draft, with no manual step in between except the one that should be manual, the final decision.

What I learned

1. Control matters more than intelligence. The gain did not come from a better model, but from limiting what the model can do.
2. Evidence you can check changes the conversation. When every claim in the diagnosis points to a file with a hash, review is fast and the model's mistakes are easy to see.
3. The draft is the right interface. A draft PR gives the team what it already knows how to use: review, history and automated tests. Nobody has to learn a new tool.
4. Stopping is a valid answer. When the evidence is insufficient or conflicting, the diagnosis becomes UNKNOWN and the run stops instead of guessing a cause. A system that can say it does not know is easier to trust.

Try it yourself

The code is at github.com/ricardoribs/pipeline-guard, under the Apache-2.0 license. If you use dbt, what would help me most is:

• running it on a test project and opening an issue about whatever broke;
• telling me which kind of failure your team hits most, so I can prioritize the next adapter;
• suggesting or testing the Airflow scenario.

I'm building in data engineering with generative AI and I'm open to conversations about opportunities. You can find me on LinkedIn.