A kernel fix leaked to a public mailing list: my triage runbook

I have had this happen on a shift: a patch that belonged on a private list lands on a public subsystem list instead. The archive picks it up in minutes, mirrors copy it, and by the time you open the thread the embargo is gone. Nothing gets un-published.

So I stopped treating these threads as gossip and started running them through a fixed procedure. On a single host it takes me roughly 45 to 60 minutes to detect, triage and get the first mitigation in place. A fleet rollout is a different animal and is really bounded by your reboot window.

Here is the path I follow, end to end.

What an embargo break really changes

An embargo is nothing more than an agreed date: details stay private until the vendors have fixes ready to ship. When a patch or bug report carrying those details goes to the wrong list, the leak is permanent. My working assumption is that the moment it hits a public archive, everything in it is public.

The important nuance is that the thread tells you where the bug lives, not whether your machines actually run that code. Keeping those two questions apart is what turns a scary mailing-list post into a short, boring list of hosts.

I subscribe a reader to linux-distros@vs.openwall.org and oss-security so the first copy I see comes from a source I chose, and I verify the archive address inside my own mail client rather than trusting a URL someone pasted into a chat.

Prerequisites and a baseline you can compare against

You need a Linux host with apt (Ubuntu 22.04/24.04 LTS, Debian 12) or dnf (RHEL 8/9, Fedora), sudo rights, the ability to reboot or schedule a reboot window, and about 2 GB free in /var — the package manager keeps the old kernel image next to the new one. I also take a VM snapshot (or lvcreate --snapshot) before touching anything.

sudo apt update && sudo apt install -y curl jq git df -h /var uname -r cat /proc/cmdline 

On the RHEL side the tooling line is:

dnf install -y curl jq git dnf-utils 

Example output, values differ per host:

Filesystem Size Used Avail Use% Mounted on /dev/vg0/var 40G 6.1G 32G 17% /var 6.8.0-xx-generic BOOT_IMAGE=/vmlinuz-6.8.0-xx-generic root=/dev/vg0/root ro quiet 

Record uname -r and /proc/cmdline in your incident notes. Without that snapshot you have no evidence that the machine actually came back up on a new kernel later. You are good to continue when Avail shows at least 2 GB and uname -r prints one release string.

Step 1: archive the thread and inventory your exposure

Read the thread, note the affected subsystem, the date and the message ID, and save a local copy. Use the URL from your own mail client.

mkdir -p ~/incident/$(date +%F) cd ~/incident/$(date +%F) curl -sS "<archive-message-url>" -o thread.html grep -o 'Message-ID:.*' thread.html | head -1 

Next, list what kernel you ship in the fleet. On Debian/Ubuntu:

dpkg-query -W -f='${Package} ${Version}\n' 'linux-image-*' | sort 

On RHEL-family:

dnf list installed 'kernel*' 

kernel.x86_64 5.14.0-362.el9 @baseos kernel-core.x86_64 5.14.0-362.el9 @baseos kernel-modules.x86_64 5.14.0-362.el9 @baseos 

Then the cheap filter that saves the most time — is the affected code even compiled in?

grep -E '^CONFIG_<SUBSYSTEM>=' /boot/config-$(uname -r) 

=y means built in, so always reachable. =m means it is a loadable module and only exposed while loaded — cross-check with lsmod. No output at all means the option is off and that code path does not exist on this host.

My triage checklist for this phase:

• Thread saved locally with date and message ID
• Affected subsystem and config option identified
• Every host labelled =y, =m, or not built
• Vendor advisory located (a CVE may only be assigned later, so check the vendor's security page directly)
• Reboot window or livepatch path agreed with the machine owner

When wc -l on your host list matches the number of machines you are responsible for, and each line carries exactly one of those three labels, this step is done.

Step 2: patch through the distro, not from mainline

Prefer the vendor package. Your distro has already backported the fix onto the kernel version you run, with the same ABI, so modules and drivers keep working. Compiling mainline changes the ABI and can break out-of-tree drivers — a second outage stacked on top of a vulnerability.

sudo apt update sudo apt-get install --only-upgrade linux-image-generic linux-headers-generic 

sudo dnf updateinfo list --all kernel sudo dnf update kernel 

The following packages will be upgraded: linux-headers-generic linux-image-generic 2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 

Verify afterwards: dpkg -l 'linux-image-*' should show ii on the new version (on RHEL, rpm -q kernel lists the new build), and /boot should hold both the new and the previous vmlinuz-* entry.

Step 3: livepatch or reboot, decide on purpose

Installing files into /boot does not remove vulnerable code from memory. Something has to activate the fix, and there are only two options.

canonical-livepatch status # Ubuntu kpatch list # RHEL-family 

If your kernel has no livepatch coverage for this fix, the reboot is not optional — plan it and do it. On RHEL-family, needs-restarting -r tells you where you stand: it prints No reboot needed and exits 0 when nothing core changed, and prints that a reboot is required to benefit from these updates with exit code 1 once a new kernel is installed.

sudo needs-restarting -r; echo "exit=$?" 

Think of it as a chain: public thread → vendor advisory → vendor kernel package → kernel image in /boot → running kernel after reboot (or livepatch module in memory). Any missing link leaves the bug reachable.

Step 4: prove the running kernel is the fixed one

uptime -s uname -r dpkg -l "linux-image-$(uname -r)" | tail -1 

2025-01-14 03:12:44 6.8.0-xx-generic ii linux-image-6.8.0-xx-generic 6.8.0-xx.xx amd64 Signed kernel image generic 

If uptime -s is not later than the package install time, you are still on the old kernel and nothing has changed. Then check for collateral damage:

journalctl -k --since "-30 min" | grep -Ei 'taint|oops|segfault' || echo "no kernel faults in window" cat /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null | sort -u 

You want no kernel faults in window, plus one line per CPU mitigation, each reading Mitigation: ... or Not affected.

All four rows matching is the real proof. And keep the previous kernel entry in /boot for one rollback cycle — if a module misbehaves, you want the way back to be a reboot, not a rebuild.

Takeaway

An embargo break is loud but not one-size-fits-all. The CONFIG_ check is the highest-value five seconds in the whole process, because it collapses a frightening public thread into the handful of hosts that genuinely need a patch. After that it is discipline: vendor package, explicit decision between livepatch and reboot, and verification against uptime -s and uname -r rather than a warm feeling that the update ran.

I keep the full write-up, including the command output I use as reference, close at hand.

This article was originally published on AI Release — https://ai-release.org — and I also post shorter notes on the channel at https://t.me/ai_release1.