A Roadmap for Improving Predictive Power, and the "If You Can't Tell, Keep Current" Rule
Re-scoring the same model on the window we actually trade showed its predictive power wasn't as near-zero as it looked. Six years of narrowing-the-universe experiments all came back "can't tell," so I left the universe alone, and I added a non-inferiority gate against a fixed anchor model for retraining deploys.
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Re-examining what "the score is low" even measures
Today started with a question: to make more money, how do I raise the model's predictive power?
Before jumping to "let's bolt on more data," I went back and looked at what that low-looking score is actually measuring.
The model learns to predict one window (one day's close to the next day's close), and it's scored on that same window.
But that's not how we trade. We buy at the next morning's open and hold for several days.
When I re-scored the same model on the window we actually trade on, the number wasn't near zero — it was noticeably higher.
So it's not that there's no predictive power. The power that exists gets largely eaten by trading costs, and what's left leans toward flagging stocks to avoid rather than stocks to buy.
So the answer wasn't "add data first"
I asked an AI to draft a roadmap for improving predictive power, and its conclusion wasn't "add new data to raise the score."
The order it recommended was: first realign what the model learns toward how we actually trade (buy in the morning, hold a few days), and only then test public data sources like disclosures and financials, one at a time, in the same frame.
I set expectations deliberately low. An improvement of this size can't be told apart from noise in a few months of live shadow trading, so the real basis for adopting anything will have to be long-horizon offline testing — and I'd rather accept that going in.
Freezing the rules before looking at any numbers
To test the first candidate (realigning the model's learning target to real trading), I wrote a pre-registration document before looking at a single result number.
I nailed down the arms to compare, the periods, the pass line, and the seeds in advance, and the commit hash for that document is itself the proof that the rules were frozen before anyone saw a result.
The reason is simple. Once you've seen the numbers, it's hard to resist nudging the pass line "in the direction that makes it pass."
This test touches no production code, no capital, and no live orders. It's a pure offline computation on CPU.
The verdict has only three values: pass / reject / can't-tell.
And "can't-tell" means keep the current setup — not "let's run it a bit longer."
Narrowing the universe — all "can't tell"
In the same sitting I ran a separate experiment: does trading a narrower universe (favoring names that trade more actively) help?
I ran six variants over a six-year window, and all six came back "can't-tell." The difference sat inside the noise.
By the rule above, can't-tell means keep current. So the universe stays as it is.
I also re-checked the opposite — widening the universe — and that too was inconclusive, so I left it untouched.
The same yardstick for retraining deploys
I added a deployment gate for when a model gets retrained and shipped.
A newly retrained model is only allowed to go live once it's confirmed to be no worse than a fixed reference (anchor) model.
And I designed it fail-closed: if non-inferiority can't be confirmed, the deploy is simply held.
It also keeps a reproducibility record so the same inputs produce the same verdict. Retraining is the quietest path for performance to leak out, so this puts one more lock on that door.
One more turn of the screw on the canonical close
Yesterday's thread about the "close price" got a follow-up today.
At the post-market re-scoring step, it now cross-checks against the confirmed bar one more time and records, every day, the rate at which it disagrees with the regular-session value from the data source(new tab).
If that rate creeps up, an alert fires. It's there to stop a quiet drift from being discovered only long after the fact.
Why this matters
Almost everything I did today was about "how do I reliably measure whether this is an improvement," not "how do I make the judgment smarter."
Honestly, an improvement of this size won't reveal itself from a few months of live trading. Telling a small edge apart from plain luck takes years.
So the decisions have to rest on frozen offline tests and conservative gates.
The hardest habit to build is letting "can't-tell" mean "change nothing." The urge to keep tinkering until something looks good always wins.
Pre-registration, three-valued verdicts, a fixed anchor, fail-closed gates — different names, but all the same move: tying your own hands before you see the numbers.