"[Oct 13] A GPU Driver Mismatch, Resolved by a Reboot With Nobody at the Keyboard"

An automatic update left the kernel module and the userspace library on different versions — recovery, from reboot to verification, ran unattended.

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

The GPU suddenly disappeared

An automatic system update rolled out a new GPU driver version this evening.

The problem was it only half-applied. The library on disk was updated to the new version, but the module already loaded in the kernel was still the old one from this morning's boot.

With the two versions out of sync, everything that needed the GPU stalled. A comparison experiment that happened to be running at the time got stuck right there.

A reboot would bring everything in line with the new version on disk. But I needed to pick a time that wouldn't collide with scheduled overnight jobs, and I had to check first whether any virtual machine or backup job would get interrupted by it.

Making the reboot itself conditional

Instead of rebooting right away, I scheduled an automatic reboot that would only fire once a set of safety conditions were met.

It checked that the virtual machine was off, that backups had finished, that it was before a certain cutoff time, and that nothing else was using the GPU at that moment. If even one condition wasn't met, it would skip the reboot and just send a notification for a human to check instead.

A reboot would also cut the working session I was in. So everything that needed to happen afterward — confirming the driver versions now matched, and resuming the experiment that had stalled — was scheduled in advance to run unattended.

In the end, I rebooted manually ahead of the scheduled time, but the verification and resume steps queued up behind it still fired correctly, so nothing double-rebooted.

Once the versions matched, I ran a GPU workload to confirm the new driver was behaving normally, and the stalled experiment picked back up.

Drawing the whole system map for the first time

The same day, I put together a single diagram laying out, for the first time, the order in which money moves through this project and what safeguards sit where.

The structure had been built piece by piece over several months, so there had never been one picture showing the whole thing at once.

I had an AI go through that diagram end to end and look for monitoring blind spots that had gone unnoticed.

A few turned up. For one, there's a spot where a scheduled pre-computation can fail and silently fall back to a different path — with no log entry and no alert when that happens. The failure itself is rare, but there was no way to notice it when it did happen.

It also surfaced that one path for pulling external market data relies on a single source with no fallback, and that backups run daily but had never actually been restored even once as a test.

Rather than fixing everything on the spot, I prioritized the list and queued it up to work through one at a time.

Why this mattered

Today's GPU incident wasn't hard to diagnose, but the key part was making sure recovery didn't need anyone at the keyboard.

Gating the reboot on safety conditions, and pre-scheduling the verification and resume steps — that mattered especially because the window ran right up against the night's scheduled production jobs.

Drawing the full system map for the first time was in the same spirit. Pulling together structure I only knew in pieces made blind spots visible that I'd normally miss.

I'll work through them one at a time from here.