On October 18, I typed this into my Claude Code session: "Normally, everything is behind the allowlist." Nothing had been deployed since Friday, but the other Claude Code session on the project, a colleague's, had pushed to master in the meantime.

Claude answered that nobody had checked yet whether every commit pushed since Friday kept to the list, and that these changes would reach every app whatever the list said:

  • new migrations, which apply to the whole database;
  • a dispatcher, a job that runs every five seconds in the production worker;
  • a new column in our proxy, which sits on the path of every app's requests.

The checklist we wrote for the next deploy, two days later, named one more: the app.js of the studio, served to every app, which no browser had loaded yet in that version. The studio is the back-office screen where you describe a section and watch its preview, and the Django service behind it.

I run backend engineering at GoodBarber. Our AI Extension Builder writes an app section from a description. Since November 4, a new engine does that work: an agent that writes the section, tests it in a real browser and fixes what it finds (the announcement). For the three weeks before, it ran in production for five test apps and nobody else. Times are Paris time, and quotes from French are my translations.

Closed by default

On October 12, I set the rule: "Let's note it as an execution rule: no impact on the apps outside the allowlist." It kept deliberate exceptions: additive migrations, measurement that changes no behavior, and dead code could still reach every app.

The list was one setting, read by one function. The next day, the agent reached production behind two more locks, on/off switches we call levers: create for creating a page and refine for changing one. They are rows in an admin table, so changing one takes no deploy. They are closed by default, and a missing row meant the old engine.

That afternoon, I reported a bug: I had asked, on a test app, for a banner on the home page, and it had landed on the page I was editing. A few minutes later, Claude answered that the agent had never run. All my test prompts had gone through the old engine, which only writes the page a prompt comes from. The list held my test app, and the levers table was empty. The answer traced it to the delivery note Claude had written for me:

The error comes from my instructions: I wrote "open create and refine", when there is nothing to open. The rows have to be added.

Claude then wrote a command to add them. It writes what the admin would, in one transaction, and refuses an app that isn't on the list, or an empty list. Without an action, it writes nothing and prints the engine each listed app gets from the routing function the views call. Once it was deployed, I ran it in production:

$ python manage.py agent_levers --open --only <five test app ids> Lever create: added, open. Lever refine: added, open. Levers: create open, refine open <app 1>: agent (create, refine) <app 2>: agent (create, refine) ... 

The old code, kept verbatim

The list also had to hold inside the code both engines share. When a shared function had to change, it checked the list: listed apps got the new body, and every other app a verbatim copy of the old one, named _legacy_<name> and marked # ALLOWLIST EXIT. A short script renamed each copy to the original's name and compared the two at a given commit, through ast.dump, so comments and formatting don't count.

On October 20, that check ran against the commit then in production. Apart from the copies added since that commit, only two differed. They had been copied from master after a change production didn't have yet, and they were inert outside the list. Until then, our todo files said to compare against master, and from then on, against the commit running in production. We then turned the script into a command that takes the commit as an argument, and a test that holds a fingerprint of every copy. Against the commit deployed that afternoon, the command reported all 70 copies identical.

The migration

On October 15, Claude told me that a new migration was on master: one table, nothing reading it yet. The third bullet of the message said that applying it through a deploy of the studio would also ship three tasks from the other session, one not reviewed yet and two with an open blocking point, and suggested waiting for the common deploy.

I had to leave, so a minute later I ran the build. The deploy reached both hosts a few minutes after that, and I applied the migration. Our migrations run from the image, and the image is built from master, so one new table took everything master held. That evening, Claude's next message opened with it: the unreviewed task was in production, and it would be reviewed first.

A migration runs once, on the whole database, and no list holds it back. Ours had to be additive, and I applied them before the code. Django lists every field of a model in its queries, so code whose model has a column the table lacks fails for every app. Old code runs fine on the new schema, because each new column is nullable or has a default in the database (db_default).

Additive still takes a lock. Several migrations went out together in the deploy that followed my allowlist message, and one added a nullable integer to the generations table, our busiest. Postgres doesn't rewrite the table for that, but the ALTER TABLE waits for an ACCESS EXCLUSIVE lock, which blocks reads too, and every query that arrives after it waits behind it. That deploy's checklist asked for a timeout, and for a second run if it fired:

docker compose run --rm -e PGOPTIONS='-c lock_timeout=5s' web python manage.py migrate 

All the migrations went through.

The scheduler

The dispatcher from Claude's answer only has work for an app on the list, but it runs on the same worker, and waits on the same queue, as every app's generations. While it has work, it queues itself again, and each run takes a worker slot from them for as long as it lasts.

Until October 22, every deploy killed that worker for every app with a SIGKILL, as I wrote in an earlier post: jobs in flight died, and rq's scheduler took minutes to come back. Since then, the worker gets its SIGTERM, but a job still running when the grace period ends still dies. The day before that fix, after a build, one host's check found no scheduler on its main queue, and the next check found it back.

The image

The studio's Config, its production settings, lives in a private submodule, linked into the image at build time. That way a config change rolls out progressively like any other deploy, and rolling back the image rolls it back too. The build script, shortened:

git checkout master git pull origin master git submodule update --init --recursive docker build --build-arg "CONFIG=${CONFIG}" -t "${IMAGE_NAME}:${GIT_COMMIT}" . 

Switching a setting on took two commits, one in the config repository and one for its pointer in the studio, then a build of whatever master held. Switching it off cost the same build. The checklist for October 21 listed 18 things to switch on, and its first point was exactly that. So master stayed frozen from the first build to the last deploy, apart from the Config pointers and one fix of ours, and the other session committed locally until we were done.

At 11:10, I asked whether everything was on. Nothing was: the morning's build had shipped the code, switched off. I wrote: "We switch everything on for the allowlist. The other session is busy, we go ahead without it." Twenty minutes later, about the Config, I told Claude: "You'll do it alone. Push, and deploy, you don't need me."

The work went in steps, with a check before and after each one, a command we had written for that day. It prints one line per point, OK, KO or MANUAL, and exits with an error if any point is KO. After the morning's build, both hosts reported the KOs we expected and a list of points to check by hand.

Fourteen of the 18 were on by 14:25, after four builds counting the morning's. Two of the fourteen needed no build, because they were levers in the admin. The checks after the last step showed no KO. The rest stayed off: three were blocked, and one by my decision.

Empty means everyone, or no one

The first setting we switched on was a fix in the Supabase connector, and that setting reads no list. The connector's gate does:

def supabase_connector_enabled_for(id_webzine) -> bool: if not getattr(settings, "FEATURE_ENABLE_SUPABASE_CONNECTOR", False): return False allowlist = supabase_connector_allowlist() if not allowlist: return True 

With the flag on, an empty list means every app. That morning, the read-only check of the production Config showed the connector's list empty, so the fix went to every app with the connector open, not to five. I had accepted that before the session, for a fix we wanted everywhere.

The agent's gate, further down the same file, has read an empty list the other way since the day I set the rule:

def agent_migration_enabled_for(id_webzine) -> bool: allowlist = agent_shadow_webzines() if not allowlist: return False 

Claude wrote it that day, in what it called a small function that follows the pattern of the existing gates, and said the inversion was deliberate: a list emptied by mistake puts every app back on today's behavior. Its docstring says it in capitals: "an EMPTY AGENT_SHADOW_WEBZINES means NOBODY". When we switched things on, four gates in that file read an empty list as everyone, and three, the agent's among them, as no one.

What I'd check before trusting a list

  • Before each deploy, list what runs for every app whatever the list says: migrations, scheduled jobs, the shared worker, static files, a proxy.
  • Read how each gate treats an empty list, and write it in its docstring and in a test. Ours split four to three in one file.
  • Print the state as the routing function answers it. Ours had the right list, an empty levers table, and an agent that never ran.
  • If your config is baked into the image and the build takes the tip of a branch, count the builds a switch-on costs, and freeze the branch meanwhile.
  • Prove the copies of the old code against the commit in production, not against your main branch.

The setting that would take the engine past the list had already spent two weeks in production, off by default. Turning it on meant one more line in that Config, and one more build of all of master. On November 4, that build carried 28 commits that production didn't have yet.