Quant Trading Bot Devlog

한국어로 보기

Price-Based Trading Rules All Failed the Backtest - Switching to an Order-Free Forward Recorder

I spent a whole day backtesting whether the price fields in my recommendation reports (entry, stop, target) could squeeze out more return, and every rule I judged fell short of the bar. So instead of putting any of them into live trading, I stood up an order-free forward recorder to judge them months from now. This is that process, plus a pre-registration rule I caught myself breaking.

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

"Isn't it a waste to just throw those price fields away?"

The reports that recommend stocks carry a few price-related fields.

Things like an entry price, a stop-loss price, a target price, and the horizon over which that target is supposed to hold.

Up to now my trading logic barely touched these values. So the starting point today was a simple question: if I pull this price information into the trading rules, would returns improve?

A full day of running them against historical data

There was only one way to answer that. Lay the rules over roughly four months of accumulated history, run them, and see whether they clear a bar I set in advance.

Rules that sell at the stop-loss price, rules that sell when the target is hit, rules that use the target to decide where to buy, rules combining both, rules that tilt position sizing by the price values.

By the end of the day I had judged more than twenty rules.

The result pointed one way. Every rule I judged fell short of the bar.

The important part is that "fell short" does not mean "lost money." I found no evidence of an edge — but no real evidence of harm either. Most of it was just noise indistinguishable from zero.

The same signal, repeated four times

Interestingly, one pattern on the buy side repeated four separate times in a single day.

Whenever I put a price ceiling on buying ("don't buy above this price"), the stocks I was blocked from buying because of that ceiling had, a few days later, risen more than the market.

In other words, the names I filtered out for "looking expensive" were actually the ones that kept running. Putting a price band on buying lost money for this one reason, every time.

The sell side looked slightly better in spots. But that gain was concentrated in a few days when the market swung hard, and in normal times it vanished. Rising-market stretches had too little data to test at all.

"Why defer a verdict I can reach right now?"

The other axis of the day was about speed.

The original design advice I'd gotten from an external AI was a two-stage structure: narrow the candidates on four months of history, then confirm months later on data that would accumulate going forward.

But once I ran things, most of them returned a "fail" right there on the historical data alone. That made "why defer until next February a verdict I can reach today?" a fair question — so every simulation I ran today was pushed to a conclusion on the spot.

I asked a separate AI whether today's simulations were done right

When you cram this many tests into one day, there's a real risk that a design drifts away from the advice's intent somewhere along the line.

So I handed the entire batch of today's simulations to a different AI — separate from the one doing the work — and asked it to independently review whether they were faithful to the original advice. I did not tell it my conclusions beforehand.

The review kept most of the five conclusions intact (no re-runs needed), but it surfaced a few problems.

The most important one was that I had broken a pre-registration rule. When computing a certain predictive-power metric, I should have honored the rule "do not look at a sealed validation window until it's over" — but part of that window leaked into the computation and even made it into the result document.

The cause was that I hadn't written the sealing constraint into the work brief. As soon as it was found, I deleted the leaked values, redacted them from the summary, and baked a date guard into the code so the same mistake can't recur.

The rest of the review was that some guard wording was more absolute than the evidence warranted. A sentence I'd written as "no effect by any measure," for instance, didn't actually have the grounds to be stated that strongly. I toned those overstatements down to match the facts.

So I decided not to ship them — and to record only, with no orders

Pulling all of this together, the decision was this.

Price-based trading rules do not go into live trading right now. The core reason is that four months of history simply lack the statistical power to resolve such a faint effect.

Instead I stood up an "order-free forward recorder."

Every day after the close, it extracts the report's price values into a fixed format and stores them, then replays seven candidate rules — not on the past, but on the data going forward — accumulating results into a separate ledger each day. Not a single real order is placed.

The key is that this is not a backtest that keeps re-examining the past, but a forward record that only accumulates on new data from today onward. An effect that no amount of squeezing the history could resolve, I'll instead test on the real market going forward, with the rules nailed down in advance.

It runs on a scheduled job that wakes itself every weekday evening, and I made it survive a reboot in the middle. I also set a reminder for mid-February next year to say "now it's time to judge." Quiet plumbing like this belongs to the operations side(new tab).

One small tweak went in here too. The recorder initially sent a success notification every time it ran, but a "ran fine" ping every weekday is just noise — so I turned off success notifications and left only the failure alert.

Why it matters

Today ended not with "I added one new rule" but with "I decided not to add more than twenty candidates."

That can feel like a waste, but shipping a rule that looks marginally good in a backtest straight into live trading is the most common trap there is. History always looks too good, and much of that "looking good" is luck.

So today's conclusion has two layers.

One is to write down an effect the history can't resolve as "can't measure yet," not as "doesn't exist." The other is to stand up the only way to actually settle it — nail the rules down in advance, place no orders, and verify only on data going forward.

Catching myself breaking a pre-registration rule for a day, only for a different AI's independent review to flag it, is the same thread. The discipline of sealing a validation window in advance is a hassle to keep, but breaking it tricks you into mistaking "looks good by chance" for "real." So scrubbing even the trace of the breach and baking in a guard mattered as much as the decision not to ship.

#quant #pre-registration #measurement discipline