"[260830] A Gate That Wouldn't Open for Months, So I Automated the Check"

An automated regime sensor, plus two overnight monitoring gaps a second AI review turned up

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

A judgment condition had been stuck for months

A few weeks ago I pre-registered an experiment to check whether a newly added AI model actually adds value to the ensemble.

The condition for reaching a verdict was that "normal" market conditions had to hold for a certain number of trading days. Checking back on it today, that condition itself hadn't been met in months.

The market had simply stayed in a high-volatility state since February.

When I originally pre-registered the experiment I optimistically assumed the condition would fill in within a few weeks — not realizing volatility was already climbing at that exact moment.

The threshold and the condition itself weren't wrong. My timeline estimate was.

Turning a manual check into an automated sensor

Up to now I'd been checking this condition by running a script by hand whenever it came to mind, which meant I had to remember to keep doing it.

So I had another AI design the implementation, and built an automated sensor around it exactly as designed.

It re-evaluates market conditions on its own at a fixed time every day, and only sends an alert once the condition is fully met.

There was already similar calculation logic elsewhere in the codebase, and reusing it directly ran into an existing safeguard — a test asserting that a research-only tool should never be called from a resident process.

So I split out just the shared calculation logic into its own module instead. That avoided duplicating the code while keeping the original safeguard intact.

I added 22 tests around the new logic, and re-running the full suite showed no impact on anything else.

Now the only moment I need to step in is when the alert fires because the condition is fully satisfied.

A second AI review turned up two gaps that weren't in the original brief

This bot runs as several separate processes(new tab), and most of its monitoring was built on the assumption that "production only runs at night."

With daytime batches becoming more common lately, I had another AI do a full review of whether that assumption still held.

Most of the points I'd originally suspected turned out to be fine, but the review surfaced two gaps that weren't in the original brief.

The first was a time-comparison check that spans midnight, where the date-boundary logic was off.

As a result, the entire second half of every night was outside the monitoring window. This was the third recurrence of a monitoring-gap incident I'd already hit before, so I fixed it right away.

The second was a process-kill rule that matched targets by model name instead of by port number.

A separate experiment happened to be using the same model name as production, which meant the rule meant to clean up production could have killed the experiment's server instead.

This was also the same class of incident as one I'd hit before, so I tightened the condition to also check the port number.

While I was at it I caught a few smaller issues too, including a safety comment left over from an earlier generation of GPU hardware.

It had been written under the assumption that a certain operation was "physically impossible" on that hardware — which was no longer true on the current setup.

I updated the comment to state clearly that the only thing standing in the way now is a single detection check for running processes, and left a note to re-review comments like this every time the hardware changes.

Live verification had to wait a day

Once the fixes were in, the plan was to run them alongside tonight's actual production batch and confirm there were no conflicts.

Then I realized today happened to be a non-trading day, so there was no overnight production run to test against in the first place.

The fixes themselves were already done and still needed regardless, so I just pushed the live verification to the eve of the next trading day.

Both pieces of work today share the same shape.

Instead of manually re-checking, over and over, when a condition will be met or whether a monitor is actually watching when it should be, today was about moving that checking itself into automation — or handing it to a second AI review to catch what manual checks kept missing.