I built a pipeline that finds open PRs in OSS repos, runs an AI review on them, and drafts a cold email to the maintainer. Sending stays manual - I press the button.

It worked. Then it didn't, and the logs were empty.

The pipeline 

scheduler (daily) → search GitHub for PRs waiting for review → filter: repo size, activity, author is human → dedup: one email per person, ever → score: "is this maintainer drowning?" → add target + run review → poll review → build draft → [human reads it, presses Send]

The last step is deliberate. More on that later.

Four loud bugs

Getting this to run end-to-end cost four bugs in a row. Each looked like a one-liner. Each only appeared after I fixed the previous one.

1. The job that didn't exist. NameError: uninitialized constant OutreachReviewJob. The Que worker loads its job list from an entrypoint script baked into the Docker image, not from the repo. I added the file, forgot the list, and the image didn't know the class existed. Fix: add to the entrypoint, rebuild the image.

2. The container that restarted forever. exec /usr/local/bin/docker-entrypoint-que.sh: no such file or directory. The file was there. The problem: I copied it from Windows, and it arrived with CRLF line endings. CRLF breaks the shebang, so the kernel can't find the interpreter. Fix: .gitattributes with *.sh text eol=lf, plus converting the file itself.

3. The API that changed. ArgumentError: wrong number of arguments (given 3, expected 1..2). Que 2.x moved delayed jobs from kwargs to job_options: { run_at: ... }. My code was written against the old API.

4. The config that drifted. One container had a required setting; the other - the one that actually ran the work - didn't. Same image, different environment. Fix: make the two environments match, and add a startup check so a missing setting fails loudly instead of at the first job.

Four bugs, four different layers: image build, file encoding, library version, container config. None of them were about the feature I was building.

The fifth bug was silent

After those four fixes, discovery ran. It found PRs, added them, started reviews. The reviews completed - I could see them in the database, findings and all.

But in the outreach panel, every target sat in reviewing. Forever. No errors in the worker logs. No failed jobs. Nothing.

Here's why. There are two jobs in the chain:

CodeReviewJob → runs the LLM review OutreachReviewJob → waits for it, then builds the email draft

The panel's "Review" button did this:

target.update(status: 'reviewing') OutreachReviewJob.enqueue(target.id) # the poller

My discovery service did this:

OutreachService.run_review!(target) # starts the review, no poller

run_review! creates the review and enqueues CodeReviewJob. It does not enqueue the poller. So the review ran, finished, and wrote its results - and then nothing happened, because the only thing that would have built the draft was never scheduled.

Nothing was broken. Nothing was watching.

The fix was one line - enqueue the poller instead of calling the service directly. But finding it took longer than any of the four loud bugs, because there was no error to grep for. The system was in a state that looked like "in progress" and stayed there.

What I took from it: when you add a second path to an operation that already exists, copy the whole chain, not the obvious first step. The button and the service were doing "the same thing" - except one of them also scheduled the follow-up. I read the service method, saw it started a review, and assumed that was the whole operation. It wasn't.

I also added a recovery task:

rake outreach:resume_stuck

It finds targets stuck in reviewing and re-enqueues the poller. Idempotent: if the review is done, it builds the draft; if not, it keeps polling. Ten minutes to write, and it would have saved an hour of staring at logs.

Two things GitHub's API taught me

stars: doesn't work in issue search. I wanted to filter PRs by repo popularity inside the search query, to avoid fetching repo info for every result. The query returned zero results. Not an error - zero. The stars: qualifier only works in repository search, not issue search. It fails silently.

GraphQL batches. Since I couldn't filter by stars in the query, I needed repo info per result - about 250 repos. REST would be 250 requests, which is how I'd been hitting GitHub's secondary rate limit (an anti-abuse limit on request density, separate from the quota). GraphQL lets you alias 40 repositories into one query:

query { r0: repository(owner: "acme", name: "widget") { stargazerCount isArchived pushedAt pullRequests(states: OPEN) { totalCount } } r1: repository(owner: "acme", name: "gadget") { ... } }

250 requests became 7. The discovery run went from over an hour to a few minutes.

The honest part: AI review is wrong about 20% of the time

I check every finding by hand before sending. On the first batch of 9 drafts, 2 contained a false finding:

  • "these annotations are evaluated at definition time" - in a file that had a future-import at the top that changes exactly that. The model didn't notice.
  • "this query adds a filter" - the filter was already there in the association.

Both were plausible. Both were wrong. That's the failure mode: not nonsense, but confident and specific.

So the pipeline stops before sending. It drafts, I read, I press Send. The automation saves the boring 90% - finding PRs, fetching diffs, running reviews, formatting emails - and leaves the judgment call to a human.

I also had to fix drafts cutting findings mid-sentence. The limit was 220 characters; real findings run 150-700. I raised it to 600, then to 2000, and made truncation prefer sentence boundaries. Any limit you set on LLM output will eventually cut the most interesting part.

What I'd tell someone building this

  • Log the "nothing happened" case. Every silent failure I've hit was a missing log line, not a missing check. If a job finishes without doing anything, say so.
  • Write the recovery task before you need it. resume_stuck took ten minutes and turned a mystery into a one-command fix.
  • Two paths to one operation is a bug waiting to happen. Extract the shared chain into one function, or accept that they will drift.
  • Keep the human where the cost of being wrong is reputation. My false positive rate is 20%. That's fine when a human reads the output. It's not fine when the output goes to a stranger's inbox.

The tool is CodeCritic - an AI code review tool. The outreach pipeline is internal, but the review Action is public: review-action.

If you've built something with a silent failure mode, I'd like to hear how you found it. Those are the bugs that don't teach you anything until you've already lost the afternoon.