3 Reasons Automated Trading Verification Tools Miss Errors

Every new verification tool I added this week failed in a different way — a bug in the tool itself, a blind spot nobody had wired a check for, and a gauge that lied about the moment it was reading

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

I've been adding several verification tools to my automated trading system lately. Each one broke in a different way when it actually got used.

One turned out to be wrong itself. One didn't exist yet, so I had no idea anything was off. One existed but read its target at the wrong moment. Here are the three cases.

1. The verification tool can have its own bug

While digging into an old question about AI model size, I went back and looked at a groundedness checker I'd built earlier.

Its job was to automatically flag report sentences that weren't backed by real data. Looking closer, it turned out to be flagging sentences that were correct but just written with a different formatting style than it expected — as "unfounded errors." The tool wasn't catching real mistakes. It was catching correct statements and calling them mistakes.

I invalidated it immediately and added a separate guard so the same kind of false positive can't slip through again.

You can't treat a verification tool as an infallible judge. The tool itself is also something that needs verifying.

2. You don't know what you're missing until you wire the check

I built a new tool to automatically check the numbers in daily reports and connected it to production.

As soon as it ran, it turned out that more than half the tickers in every daily report had formatting errors. Looking back 45 days, the same class of error had been there every single day without exception.

There had already been a self-consistency check on report numbers before this. But it was structurally blind to this particular class of error — it simply couldn't see it. "We have a check" and "we have a check for this specific error" turned out to be two different sentences.

Until the new tool was wired in, nobody knew this error existed. It was another case of "the system is running, but nothing is actually being verified."

Believing you have verification isn't enough. You have to explicitly confirm what class of error that verification actually covers.

3. If the read timing doesn't reflect real state, the gauge fools itself

There's a gauge that automatically checks model reproducibility every day. It died two nights in a row.

The cause: another overnight job was holding onto the GPU, but the gauge decided "it's free right now" based on a single instantaneous read. Starting up under that false assumption failed every time.

I switched it to directly check occupancy instead. Instead of loading a fresh instance each time, it now takes over the one the overnight job was already using right as that job wraps up — removing the failure point at its root.

If the way a gauge reads "current state" competes for the same resource it's trying to measure, that reading can't be trusted. A snapshot in time and actual occupancy are two different questions.

Putting the three together

All three cases showed the same thing: "we built a verification tool" and "we are actually being verified" are different sentences.

Building the tool wasn't the end of the work. I still had to keep asking whether the tool itself could be wrong, whether some error class was invisible to it from the start, and whether its read method could collide with the thing it was measuring.

Every time I bolt on a new verification tool now, "what is this thing missing right now" goes on the list as the next thing to check.