The incident that never shows up in your deploy log

Here is a scenario that's easy to create on Gemini Enterprise Agent Platform today.

You build a Managed Agent. You attach a skill from Skill Registry called invoice-triage. It works great. You ship it.

Two weeks later, someone on another team updates invoice-triage to handle a new vendor format. They test it with their own agent. It works for them.

That same afternoon, your production agent starts misrouting invoices.

Nobody touched your agent. No pull request, no deploy, no change ticket. Your CI pipeline is green. Your agent config in Git is identical to last week.

What changed is the skill underneath it. I call this skill drift, and in this post I will show you exactly why it happens, how to measure a second hidden problem that comes with it (the context tax of mounting many skills), and a small lockfile pattern that fixes both.

Everything here is based on the public Managed Agents API and Skill Registry docs. Both are Preview, so treat the code as a starting point and test it in a sandbox project first.

Quick refresher: how skills reach your agent

A skill is a folder with a SKILL.md file plus optional scripts/, references/, and assets/. You upload it to Skill Registry as a zipped payload, and the registry keeps a revision history for every update.

When you create or update a Managed Agent, you mount skills into the agent's sandbox through base_environment.sources. The docs give you three ways to point at the registry:

Source path What the agent gets
projects/P/locations/L/skills/SKILL_ID The default version, whatever it is right now
projects/P/locations/L/skills/SKILL_ID/skill_versions/VERSION One exact version, frozen
projects/P/locations/L/skills Up to 100 skills from that project and location

The first and third options are the ones people copy from quickstarts. Both are floating references. That is the root of the problem.

Problem 1: floating references are :latest tags for agent behavior

We learned this lesson years ago with container images. Deploying myapp:latest to production means your production state is defined by whoever pushed last, not by what you reviewed.

A skill path without a version is the same thing, except worse in one way: a skill is mostly natural language instructions. A tiny wording change in SKILL.md can change which tool the model calls, what it refuses, or how it formats output. There is no compiler error. There is no failing unit test unless you wrote evals for it.

A few details from the docs make this sharper:

  • UpdateSkill can replace the display name, description, and the whole zipped filesystem in one call.
  • The skill ID becomes the folder name inside the agent sandbox, so the agent keeps "seeing" the same skill name even after the content changed.
  • Skill Registry inherits project level IAM. Anyone with write access in the project can update any skill in it.

So the blast radius of one skill update is every agent that mounts that skill without a version.

Problem 2: the context tax of "mount everything"

The third path above is tempting. One line, and your agent has every skill in the project.

But think about how skills work. In the Agent Skills pattern, the agent sees each skill's name and description up front so it can decide which one to open. The full SKILL.md body only loads when the skill is picked. That is called progressive disclosure, and it keeps things cheap when you have a handful of skills.

With up to 100 skills mounted, those names and descriptions add up. That text rides along on every turn, whether or not any of those skills are relevant.

There is also a quality cost. More skills with overlapping descriptions means more chances for the model to pick the wrong one. Two skills named invoice-triage and invoice-review with similar descriptions is a routing bug waiting to happen.

You cannot manage what you do not measure, so let's measure it.

Measuring the skill metadata budget

This script lists every skill in a project and location, joins the names and descriptions the way an agent would roughly see them, and counts tokens with the Count Tokens API.

# skill_context_tax.py import os import requests import google.auth from google.auth.transport.requests import Request from google import genai PROJECT = os.environ["GCP_PROJECT_ID"] LOCATION = os.environ.get("SKILL_LOCATION", "us-central1") MODEL = os.environ["MODEL_ID"] # the Gemini model your agent uses BASE = f"https://{LOCATION}-aiplatform.googleapis.com/v1beta1/projects/{PROJECT}/locations/{LOCATION}" def authed_session(): creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"]) creds.refresh(Request()) s = requests.Session() s.headers["Authorization"] = f"Bearer {creds.token}" return s def list_skills(s): skills, token = [], None while True: params = {"pageToken": token} if token else {} r = s.get(f"{BASE}/skills", params=params) r.raise_for_status() body = r.json() skills.extend(body.get("skills", [])) token = body.get("nextPageToken") if not token: return skills def main(): s = authed_session() skills = list_skills(s) catalog = "\n".join( f"- {sk.get('displayName', '')}: {sk.get('description', '')}" for sk in skills ) client = genai.Client(vertexai=True, project=PROJECT, location="global") total = client.models.count_tokens(model=MODEL, contents=catalog).total_tokens print(f"Skills in {LOCATION}: {len(skills)}") print(f"Metadata tokens per turn (floor estimate): {total}") if skills: print(f"Average per skill: {total / len(skills):.1f}") if __name__ == "__main__": main()

Two honest caveats about this number:

  1. It is a floor. The base agent may wrap each skill in extra formatting, file paths, or instructions that you cannot see.
  2. It uses the registry description. If the description in your SKILL.md frontmatter is longer, the real cost is higher. Keep the two in sync.

To turn it into a monthly number, multiply by your turns per month and your model's input price. Here is the math with made up example numbers, just to show the shape:

80 skills x 70 tokens = 5,600 tokens of skill metadata per turn 5,600 x 2,000,000 turns/month = 11.2 billion input tokens per month

Plug in your own skill count, turn volume, and the current price from the pricing page. If the result surprises you, that is the point of measuring.

The fix: a skill lockfile

The pattern is the same one we use for package managers. Floating references are fine for development. Production gets exact, reviewed versions recorded in a file that lives in Git.

The flow looks like this:

author updates skill -> registry creates new revision -> staging agent (floating) picks it up -> evals run against staging -> pass? update skills.lock.json in a PR -> merge -> CI pins prod agent to locked versions

Step 1: generate and check the lockfile 

# skill_lock.py import json import os import sys import requests import google.auth from google.auth.transport.requests import Request PROJECT = os.environ["GCP_PROJECT_ID"] LOCATION = os.environ.get("SKILL_LOCATION", "us-central1") BASE = f"https://{LOCATION}-aiplatform.googleapis.com/v1beta1/projects/{PROJECT}/locations/{LOCATION}" LOCKFILE = "skills.lock.json" def authed_session(): creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"]) creds.refresh(Request()) s = requests.Session() s.headers["Authorization"] = f"Bearer {creds.token}" return s def latest_revision(s, skill_id): revs, token = [], None while True: params = {"pageToken": token} if token else {} r = s.get(f"{BASE}/skills/{skill_id}/revisions", params=params) r.raise_for_status() body = r.json() revs.extend(body.get("skillRevisions", [])) token = body.get("nextPageToken") if not token: break if not revs: return None revs.sort(key=lambda rev: rev["createTime"]) newest = revs[-1] return {"revision": newest["name"].split("/")[-1], "created": newest["createTime"]} def lock(skill_ids): s = authed_session() data = {"project": PROJECT, "location": LOCATION, "skills": {}} for sid in skill_ids: data["skills"][sid] = latest_revision(s, sid) with open(LOCKFILE, "w") as f: json.dump(data, f, indent=2) print(f"Wrote {LOCKFILE} with {len(skill_ids)} skills") def check(): s = authed_session() with open(LOCKFILE) as f: data = json.load(f) drift = [] for sid, locked in data["skills"].items(): current = latest_revision(s, sid) if current is None or current["revision"] != locked["revision"]: drift.append((sid, locked["revision"], current and current["revision"])) for sid, old, new in drift: print(f"DRIFT {sid}: locked {old}, registry now has {new}") sys.exit(1 if drift else 0) if __name__ == "__main__": if sys.argv[1] == "lock": lock(sys.argv[2:]) elif sys.argv[1] == "check": check()

Usage:

python skill_lock.py lock invoice-triage vendor-lookup refund-policy python skill_lock.py check # exits 1 if any skill moved

The check command is your drift alarm. It does not block anyone from updating a skill. It just tells you, loudly, that the registry no longer matches what production was approved with.

Step 2: pin the production agent to the lockfile

The agent update uses PATCH with update_mask=base_environment. There is no update method in the Python SDK for agents yet, so this goes through REST.

One gotcha worth calling out: the docs examples resend the whole base_environment object, including the network allowlist. Treat this update as a full replace, not a merge. If you send only sources, plan for the possibility that other environment settings go back to defaults. Network access is off by default, so a lost allowlist can break an agent that calls external APIs.

# pin_agent.py import json import os import requests import google.auth from google.auth.transport.requests import Request PROJECT = os.environ["GCP_PROJECT_ID"] AGENT_ID = os.environ["AGENT_ID"] # The agent docs call this segment "skill_versions", while the registry # returns revision names under "revisions". Confirm which one your project # accepts, then set it here. PIN_SEGMENT = os.environ.get("PIN_SEGMENT", "skill_versions") ALLOWLIST = json.loads(os.environ.get("NETWORK_ALLOWLIST", '[{"domain": "api.example.com"}]')) AGENT_URL = ( f"https://aiplatform.googleapis.com/v1beta1/projects/{PROJECT}" f"/locations/global/agents/{AGENT_ID}?update_mask=base_environment" ) def main(): with open("skills.lock.json") as f: lock = json.load(f) sources = [] for sid, info in lock["skills"].items(): path = ( f"projects/{lock['project']}/locations/{lock['location']}" f"/skills/{sid}/{PIN_SEGMENT}/{info['revision']}" ) sources.append({"type": "skill_registry", "source": path, "target": "/.agent/skills"}) body = { "name": AGENT_ID, "base_environment": { "type": "remote", "sources": sources, "network": {"allowlist": ALLOWLIST}, }, } creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"]) creds.refresh(Request()) r = requests.patch( AGENT_URL, headers={"Authorization": f"Bearer {creds.token}", "Content-Type": "application/json"}, json=body, ) r.raise_for_status() print("Update started:", r.json().get("name")) if __name__ == "__main__": main()

The update is a long running operation, so poll the returned operation name before you call the rollout done.

Step 3: run the drift check on a schedule

Any scheduler works. Here is a minimal GitHub Actions version that runs every morning and fails loudly on drift:

name: skill-drift-check on: schedule: - cron: "0 12 * * *" workflow_dispatch: jobs: check: runs-on: ubuntu-latest permissions: id-token: write contents: read steps: - uses: actions/checkout@v4 - uses: google-github-actions/auth@v2 with: workload_identity_provider: ${{ secrets.WIF_PROVIDER }} service_account: ${{ secrets.DRIFT_SA }} - run: pip install requests google-auth - run: python skill_lock.py check env: GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }} SKILL_LOCATION: us-central1

The service account only needs read access, so roles/aiplatform.viewer is enough for this job.

A few more things I noticed while building this

Skill IDs are a namespace you cannot get back quickly. Skill IDs are immutable, and after a delete the ID stays reserved for a while (the docs mention 24 hours in one place and permanent reservation in another, so plan as if it is long). Pick names carefully, and do not use delete plus recreate as a "rollback" strategy. Pinning a previous version is the rollback.

Regions do not line up one to one. Managed Agents only support the global location, while Skill Registry lives in specific regions (us-central1, europe-west4, us-east5 at the time of writing). Your mount path includes the skill's region, so write that region into your lockfile instead of assuming it.

Cloud Storage mounts skip validation. You can also mount skills straight from a bucket. The docs say the backend does not validate folder content in that case. That makes the registry path the safer choice for production, because uploads go through payload checks and you get revision history.

Write descriptions for routing, not marketing. Since descriptions are both your token cost and the model's routing signal, short and specific beats long and friendly. "Classifies AP invoices by vendor and urgency. Not for refunds." does more work than a paragraph.

Limitations and caveats

  • Skill Registry and Managed Agents are Pre-GA. APIs, paths, and field names can change, and the docs say not to use confidential data with the Managed Agents preview.
  • At the time of writing, Skill Registry does not support VPC Service Controls, CMEK, or HIPAA. Check the compliance table before you design around it.
  • The docs use two different names for a pinned skill path (skill_versions on the agent side, revisions on the registry side). My script makes this configurable because I could not confirm which is final. Verify in your own project.
  • The token numbers from the context tax script are a lower bound, not an exact bill. The base agent may add text you cannot see.
  • A lockfile tells you what changed. It does not tell you whether the change is good. Pair it with evals on a staging agent before promoting.

Wrapping up

Skills make agents easier to build and share across teams. That same sharing means a skill update is a production change for every agent that mounts it without a version.

Three habits fix most of it: pin versions in production, measure what "mount everything" costs you, and run a daily drift check against a lockfile in Git. None of this needs new infrastructure. It is the same discipline we already apply to container images and package managers, pointed at a new kind of artifact.

If you try this pattern, I would love to hear what your skill metadata token count came out to.

References