Most teams can name the package registry they depend on. Far fewer can say how many times their CI reaches for github.com directly. We counted ours this morning, and the trigger was not hypothetical.
On November 17, github.com/microsoft/onnxruntime started returning 404. No notice, no archive banner, no redirect. We checked it ourselves at 10:15 Beijing time on November 18 and again two hours later: still 404. The thread on Hacker News filled up the same day, and the first comment set the tone: a team whose evals started failing in their pipelines that morning, with no advance warning they could find.
Here is the part that decides what you should do about it. The PyPI package is still up. So are the docs at onnxruntime.ai and the NuGet package. What vanished is the repository itself: source, submodules, release assets, issue history, the security discussions people link to in audits. If your stack only ever touches the wheel, you felt nothing. If anything in your build clones, pins, or downloads from that repo, it broke on Monday.
Upstream risk is the chance that a repository, model, or artifact source you build on goes offline with no notice, no export window, and no contract that says otherwise. Microsoft has not said why the repo is gone, and it may well come back. That changes nothing about the audit below, because the next one to vanish will not announce itself either.
The audit: five passes over your repo
Run everything from the repo root. Each pass answers one question: where does my build trust someone else's hosting? For each, you get the command, what the output means, and what to do with a hit. Budget an hour for a mid-sized service, most of it spent reading, and keep the results; the mirror section below works straight off this list.
Pass 1: lockfiles
grep -n "github.com" package-lock.json yarn.lock pnpm-lock.yaml \ requirements*.txt poetry.lock Pipfile.lock go.sum Cargo.lock 2>/dev/null What it catches: packages installed from a repository instead of a registry. Registry entries resolve to the registry, which holds its own copy of every published artifact. Anything matching here resolves to a repo. In Python lockfiles, look for git+https lines; in JS lockfiles, resolved: "git+..." or a github.com tarball URL; in Go, go.sum entries are proxied by default through proxy.golang.org, so a hit in go.mod with a replace directive pointing at GitHub matters more than the sum file itself.
What to do with a hit: decide, per line, whether the package has a registry release you can switch to. Many git-pinned dependencies exist only because someone needed one unreleased fix years ago, and the fix has long since shipped in a tagged version. Switching those back to the registry is the cheapest de-risking you will do all week. Verify the swap the boring way: build both, diff the installed tree (pip download the tagged version and compare against what the git pin produced, or diff node_modules listings), and run the test suite before merging. The remainder, the ones that genuinely live only in a repo, go on the mirror list.
Pass 2: submodules
git config --file .gitmodules --list 2>/dev/null What it catches: source trees fetched fresh on every clone. Every URL in that output is resolved at checkout time, which includes the checkout your CI does on each run. A vanished submodule breaks every new clone of your own repository, which is a strange and unpleasant failure: your repo is fine, your history is fine, and no fresh machine can build it.
What to do with a hit: submodules are the strongest mirror candidates because git makes relocating them painless. Mirror the upstream (next section), then point the submodule URL at your mirror and keep the upstream as a comment:
git submodule set-url third_party/somelib https://git.internal/mirrors/somelib.git Commit hashes do not change, so nothing else in your history moves.
Pass 3: Dockerfiles, Makefiles, and scripts
grep -rn "github.com.*releases/download\|raw.githubusercontent.com\|git clone" \ Dockerfile* Makefile makefiles/ scripts/ 2>/dev/null What it catches: release tarballs and raw-file downloads baked into images and setup scripts. Dockerfiles love a curl -L straight into a release asset, and base-image build scripts love piping an installer from a raw URL. These URLs return 404 the moment the repo does, and they fail at image build time, which often means they fail during an incident, exactly when you are rebuilding something.
What to do with a hit: download the asset once, store it in your artifact store (the same place your own build outputs go), and change the Dockerfile line to pull from there. Record the original URL and the file's checksum next to it, so provenance survives:
curl -LO https://github.com/ORG/REPO/releases/download/v1.2.3/tool.tar.gz sha256sum tool.tar.gz >> MIRROR_MANIFEST.txt Pass 4: CI workflow actions
grep -rn "uses:" .github/workflows/ 2>/dev/null | grep -v "^\s*#" What it catches: the pass teams forget. Each uses: someorg/some-action@v4 line is a repository someone else hosts, fetched at run time, executed with your credentials. Pinning to a commit SHA protects you from tampering. It does not protect you from deletion; the SHA still has to be fetched from a repo that exists.
What to do with a hit: triage by blast radius. Actions maintained by your CI vendor are about as durable as the CI platform itself; if those vanish, you have bigger problems, so leave them. Third-party actions from individual maintainers are the fragile tier. For each one, either vendor it (most actions are a few hundred lines of JavaScript or a composite YAML you can copy into .github/actions/ in your own repo) or fork it into your own org and point uses: at the fork. Vendoring is usually less work than it sounds:
mkdir -p .github/actions/some-action # copy the action's action.yml and dist/ into it, commit, then: # uses: ./.github/actions/some-action Pass 5: model weights and AI-stack extras
grep -rn "huggingface.co\|cdn-lfs\|\.onnx\|\.gguf\|\.safetensors" \ --include="*.py" --include="*.yaml" --include="*.toml" . 2>/dev/null | head -50 What it catches: the heaviest upstream dependency most AI teams have, and the most casually fetched. Weights are tens of gigabytes, so people deliberately avoid keeping second copies, and the download URL quietly becomes the backup. Hosting policies for model files change more often than for code: several model families have moved behind license-acceptance gates after release, which breaks anonymous CI downloads even while the files nominally stay up.
What to do with a hit: copy the exact files you serve, in the exact quantization you serve, to storage you control, and make the serving config read your copy first. A second disk is enough. While you are there, sweep the small companions that travel with weights and break loading when absent: the tokenizer files, the generation config, the chat template. They are kilobytes, they live in the same upstream repo as the weights, and a surprising number of serving stacks fetch them separately at startup. Copy the whole model directory rather than the big file alone. The download script that fetched them originally becomes the refresh script, pointed at your copy as the primary and the public URL as fallback.
The tally
grep -rhoE "github.com/[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+" \ --include="*.json" --include="*.lock" --include="*.yml" --include="*.yaml" \ --include="*.toml" --include="*.txt" --include="Dockerfile*" . 2>/dev/null \ | sort -u One line per unique repo your build trusts. Read the list slowly. Every entry is a thing that can go 404 on a Tuesday without asking you.
One more pass decision worth writing down: what not to mirror. Transitive registry packages are already buffered by the registry and, if you set up the wheelhouse or proxy below, by your cache too; mirroring their repos adds nothing. Build outputs you can regenerate from your own source do not need copies, they need the recipe. And the long tail of small tools whose loss would cost you an hour of finding a replacement is not worth a line in the manifest. The test for every candidate is the same: if this vanished tonight, would the business stop, or would someone sigh and move on? Mirror the first category. Sigh at the second.
The mirror: one afternoon
A mirror is a copy you control, refreshed on a schedule. For the repos that survived triage:
# one-time, per repo git clone --mirror https://github.com/ORG/REPO.git /mirrors/REPO.git A --mirror clone carries every branch, tag, and ref. Two things it does not carry, and both have bitten people who thought they were covered:
One gap is Git LFS objects. A mirror clone copies the LFS pointer files, which are tiny text stubs, and leaves the actual binaries on the LFS server. If the upstream uses LFS (model repos often do), fetch the real objects explicitly:
cd /mirrors/REPO.git && git lfs fetch --all The other gap is release assets. Releases are attached to the repo in GitHub's database, outside git's object store entirely. A mirror clone knows the tag a release points at and nothing about the tarballs attached to it. Grab those separately; note that the API paginates, so for a project with a long release history loop until empty, or let the CLI do it per release:
gh release list --repo ORG/REPO --limit 200 | awk '{print $1}' | while read tag; do gh release download "$tag" --repo ORG/REPO --dir "/mirrors/assets/REPO/$tag/" || true done Then the refresh, which is the part that makes it a mirror rather than a snapshot:
# /etc/cron.d/mirror-refresh (or a scheduled CI job) 15 3 * * 1 mirror for d in /mirrors/*.git; do git -C "$d" remote update --prune; done Weekly is a sane default. The refresh is incremental, so after the first clone it costs seconds per repo. If you mirror more than a handful of repos, run the refresh with an authenticated token rather than anonymously: unauthenticated clone and API traffic hits rate limits quickly, and a refresh job that half-completes on throttling gives you mirrors of uneven age, which is worse than mirrors of known age. The read-only token from the credentials note below is the right one to use here.
For registry packages, the equivalent move is a local package store. Python's is built in:
pip download -r requirements.txt -d ./wheelhouse # restore path: pip install --no-index --find-links=./wheelhouse -r requirements.txt For Node, a pull-through proxy such as Verdaccio caches every package your builds request and keeps serving them if the upstream entry disappears. For containers, most registries (Harbor, Artifactory, even a plain registry:2 with a proxy config) can run as a pull-through cache for Docker Hub and ghcr.io. While you are in the Dockerfiles anyway, pin base images by digest (FROM python:3.12-slim@sha256:...) rather than by tag: a tag is a moving pointer on someone else's registry, and a digest plus your pull-through cache together mean the image you built with is the image you can always rebuild with. The common shape across all three ecosystems: your build asks your cache, your cache asks the internet once, and from then on the internet is an optimization.
Where to put the mirrors is less important than having them. A NAS in the office works. So does a persistent disk on the box that runs your CI. One useful rule: put the mirror in a different failure domain from the thing it protects. A mirror that lives in the same cloud account as your CI protects you from upstream deletion and from nothing else; a disk in the office additionally survives your cloud account having a bad day. The storage for a typical service's full dependency set is measured in tens of gigabytes, which is to say: less than one model checkpoint.
Pointing CI at your copies
A mirror nobody reads is a warm feeling, so the last step is making the build prefer it. Doing this incrementally beats a big-bang migration, because each step is testable on its own.
Order the work by blast radius: start with the dependency whose disappearance would stop you longest (for most AI teams that is weights, then the inference runtime, then submodules), and leave the long tail of small packages to the registry cache.
The fallback order inside each build step should read: your mirror first, public URL second, fail third. In a Dockerfile that is usually two lines and an ||; in CI config, most runners let you define the mirror as the primary source and the public host as a named fallback. Resist the temptation to flip it (public first, mirror as backup): that configuration looks the same on a green day and silently rots, because the mirror path never gets exercised until the day it must work.
Which leads to verification. You validate a mirror the same way you validate offline operation: cut the path and watch. Add a CI job, weekly is plenty, that runs the build inside a network namespace where github.com and the registry hosts are unreachable, and passes only if the build completes from mirrors alone. On most runners the cheapest version is a container with an /etc/hosts that pins the public hosts to a blackhole address. The first run of that job is usually educational: it finds the two or three fetches every team forgot they had.
Checksums close the loop. Keep a MIRROR_MANIFEST.txt that maps each mirrored artifact to its sha256 and the original URL, and have the refresh job verify it. If an upstream force-pushes or re-tags (it happens), you want the refresh to fail loudly rather than quietly absorb the rewrite, because a silent change in mirrored content is exactly the class of thing the security team should look at before the build does. Weights deserve the same treatment with higher stakes: hash the exact files you serve when you first copy them, and have the serving host verify the hash at load time. A corrupted or swapped weight file does not fail loudly on its own; it just answers differently.
Two small operational notes that save grief later. Give the mirror host its own read-only credential story: CI should be able to fetch from mirrors with a token that can do nothing else, and the refresh job's write credential should live on the mirror host only, never in CI secrets, so a compromised pipeline cannot rewrite your copies. And write the three-line README in /mirrors/ saying what refreshes when and who owns it. Mirrors fail silently when the one person who set them up leaves; the README costs a minute and outlives them.
What a team actually does with this
Consider a ten-person team running a product with a retrieval service, two fine-tuned models in production, and ordinary CI. Their audit produces: 3 git-pinned Python packages (two have tagged releases now, one is genuinely repo-only), 2 submodules, 6 third-party actions (4 from the CI vendor, 2 from individuals), a Dockerfile that curls one release tarball, and 160 GB of weights fetched from a hosting URL at deploy time.
The afternoon's work: switch two lockfile lines back to the registry, mirror the third repo plus both submodules (three git clone --mirror commands, one LFS fetch), vendor the two individual actions, move one tarball into the artifact store, and copy the weights to the NAS with the serving config reading the NAS path first. The cron entry and the weekly blackhole job go in before anyone forgets. Total new storage: under 200 GB, dominated entirely by the weights. Nothing about the team's daily workflow changes; the public internet is still the convenient path, just no longer the only one.
That is the whole program. It is not elegant, and it does not need to be.
The week after, the useful move is making the default stick so the list never regrows. Put the mirror-first fallback into whatever template new services are stamped from, so the next repo starts with the right fetch order instead of inheriting the audit as a chore. Add the tally command to the new-service checklist, and have the weekly blackhole job cover every service, including the ones that never got the initial treatment. From then on the audit stops being an event; it is a property the template enforces and the weekly job proves.
One honest caveat before the product section. A mirror addresses upstream risk and nothing else. If the upstream shipped a backdoored commit last month, your mirror preserves it faithfully; that is security risk, and it needs review, not copies. And a mirror of an abandoned project still receives no fixes; that is maintenance risk, and at some point it needs a fork with an owner or a migration. The weekly refresh even gives you a natural checkpoint for noticing the second kind: a mirror that has pulled nothing new for six months is telling you something about the upstream's health. Keep the three risks apart when you decide what deserves the effort.
How we apply this one layer up
Our own product, KarmaBox, is built on the other side of that bet: the runtime, the models, and the avatar's memory live on a machine you own. The device ships with its models preinstalled, so the day-to-day work an AI avatar does there, chat, writing, retrieval, image understanding, transcription, video, runs without reaching upstream at all. The local loop is enforced: if a local job fails, it reports failure rather than quietly falling back to a cloud endpoint, which is the same design argument as mirror-first fallback order above. In a three-day unplugged test we ran in October (October 28 to 30, on our own device), the box logged 197 local inferences with zero business data leaving it. The avatar's memory sits in a file the owner can carry to another machine. When a job genuinely needs a cloud model, that path is built in, with no keys for the owner to manage, and anything that costs money asks for approval first.
The audit above is the same idea at a smaller scale: know what you pull from other people's machines, and hold a copy of whatever you cannot afford to lose.
The repo may be back by the time you read this. Suppose it is. How many entries on your tally would survive the next one, and which of them would you miss first?
Sources
Hacker News, Microsoft's ONNX Runtime has disappeared from GitHub, November 17, 2026
PyPI, onnxruntime, checked November 18, 2026
Microsoft, ONNX Runtime documentation, checked November 18, 2026

![When is the memory location of `Box<[u8]>` guaranteed to remain stable?](https://emoji.discourse-cdn.com/twitter/wink.png?v=16)

