"[260829] Switching to a New Local Model, and Quietly Keeping the Old One Around"

I wrapped up weeks of model comparison and made the switch official, then caught a completion-detection bug in a separate experiment

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

Weeks of model comparison, and today I made the switch official

This bot uses a locally hosted LLM to analyze stocks.

For a while now I've been comparing the model currently in production against a candidate with a different architecture (a dense model in the 27B range).

The overnight production batch has been running clean, and a consistency metric I track across repeated runs on the same stock set landed back in the normal range I'd already established.

So today I made the switch to the new model official.

While I was at it, I also fixed a piece of stale code.

The alert message that reports which model is currently active had the old model's name hardcoded, so even after switching models it kept pointing at an already-retired model as "production."

I changed it to read the currently active model dynamically, so the next model switch won't require touching this again.

Keeping the retired model around as a zero-weight shadow

Right after the switch, a new idea came up.

Instead of dropping the retired model entirely, why not keep it running alongside the new one inside the AI ensemble pipeline(new tab) that both models feed into?

Rolling this straight into production weighting felt risky, so I had another AI review the proposal.

The verdict came back conditional.

The model switch had only just landed, and other validation work was still in flight — too many changes overlapping at once, was the concern.

So instead of applying it live, we started by adding the retired model back in at zero weight, purely to log its output.

At zero weight it has no effect on actual rankings — it just quietly accumulates data on how closely the two models agree on the same stocks.

GPU memory won't fit both models resident at once, so the setup scores the full set with the new model first each night, then re-scores the same stocks with the retired model right after.

In case anything goes wrong along the way, I made sure the process always falls back to the production model at the end, so the live service can't end up silently stuck on the wrong one.

Once a few weeks of data build up, I'll decide whether to actually use this pairing, and if so, how much weight to give it.

A completion-detection bug caught in a separate experiment

Around the same time, a repeated-run validation job I had going kept finishing suspiciously fast — just a few seconds.

The log said "done," so at first I assumed it had actually finished.

It turned out the previous run had stopped partway through, and on restart the job wrongly treated "some results already saved" as "fully done," so it exited immediately instead of resuming.

The bug was checking whether results existed at all, not how many stocks had actually been covered.

Once I added proper logic to check full completion, the job correctly picked up and finished the remaining stocks.

The blog's own publishing cadence changed too

I'd been setting up a feature to automatically post a recurring performance report on this blog, but decided that publishing it daily would expose the actual trading rhythm in too much detail, so I switched it to a much less frequent cadence.

Since the cadence is dropping anyway, I also decided to publish the same report to the other channel where I share this project.

The two things that stood out most today were the model switch and the shadow observation setup that followed it right away.

The reason for keeping the old model alive at zero weight instead of shutting it off outright is simple.

Watching with zero impact until the data builds up is a lot safer than deciding without any data at all.