Quant Trading Bot Devlog

한국어로 보기

Fixing the Measurement, Then Redefining the Goal - Absolute Return and Execution-Layer Cleanup

Clearing away a benchmark illusion showed the stock-selection edge was essentially zero, the owner locked the system's goal to roughly 10% absolute return per year, and the "always fully invested" principle was scrapped. The rest of the week went to the intraday filter, order retries, and measurement discipline that decision demanded.

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

The question that ran through the week

This week the days didn't stand apart — they formed a single thread.

It started with "how do we measure performance," and once the measurement was fixed, a decision followed about what this system is even for.

That decision, in turn, drove the execution-layer and measurement-discipline work for the rest of the week.

The question running through all of it was one: is the number in front of me real, or is it an illusion or just noise?

First, fix the measurement

Monday's starting point was that the "vs. index" line in my live-account performance tracking report(new tab) had been printing badly negative in a falling market.

I handed the root-cause analysis to another AI tool, and the cause narrowed down to an illusion: the numerator and denominator were measured over different time spans.

"Total managed return" was a two-month cumulative, while the "index" it was compared against only measured from the point each currently-held name was bought. In a down market that span mismatch widens structurally.

So I redefined that line entirely as a stock-selection effect — an apples-to-apples comparison of each lot against the index move over exactly the same holding period.

Measuring honestly gave a more uncomfortable answer. The stock-selection edge was essentially zero, and nearly all of the loss came not from the names but from the whole market falling (beta).

That exposed a contradiction. "Stay positive even when the index drops" and "never leave cash idle, always stay fully invested" simply cannot both hold.

A long-only book that's always fully invested has nothing to do but fall along with the market. Choosing between the two wasn't a matter of tuning the signal — it was a decision.

Then, redefine the goal

On Tuesday, that decision came. This was the week's turning point.

First the owner nailed down what the system is: not a research toy but a system for making money, with success measured not against the index but by absolute return.

Clearing roughly 10% a year over the long run, after all trading costs including electricity, counts as success. Not that exact number every year — a compounded long-run average near it is enough.

The same decision scrapped an old principle: "never leave cash idle."

Cutting exposure and holding cash when the market is bad is no longer a violation of principle but a normal option.

A distinction stays, though. Cash the signal deliberately chose to hold is fine; cash that piles up unintentionally from a config error is still a defect.

Redefining the goal the very day after fixing the measurement was no coincidence. Without measuring honestly, there would have been no basis to make this decision at all.

The execution-layer work the new goal demanded

Once the goal became "cut costs and lose less in a down market," the immediate lever was reducing trading friction.

The first move was adding an intraday trade filter(new tab) to the live account. If a name I meant to buy spikes intraday it defers the buy; if a name I meant to sell drops sharply it defers the sell.

The first version had several scary bugs — double-counting sell credits, pending orders offsetting each other, reverse-direction reorders. This is the path real money goes out on, so it went live only after a review killed those.

But within a day a more fundamental flaw surfaced. The reference value the filter uses to judge spikes and drops — the previous day's close — was itself wrong.

One portal's definition of the daily-candle "close" had, at some point, started including after-hours trading, so the value pulled from the market-data source(new tab) differed subtly from the exchange's regular-session close.

So I corrected the reference to the brokerage API's regular-session close and preregistered that v1.1 for a two-month validation window. The first version, running on a corrupted reference, had no standing to be judged and was closed as indeterminate.

The same day brought a different kind of incident. While running the regression tests, six real notifications went out.

One notification path and the new filter's reference-value lookup weren't caught by the mocking layer. I globally blocked both in the test environment and reconfirmed the tests pass.

Wednesday smoothed the remaining rough edges in the execution path. Morning first-round buys were getting rejected for "insufficient buying power" because of a timing gap in sale proceeds about to arrive; I made those retry exactly once within the same round.

Retry orders carry a distinguishing tag to prevent a duplicate buy. Orders left unfilled all day on a stale planned price now get cancelled just before the next round and replanned at the current price.

Cancellation is a more dangerous action if done wrong, so I had another AI review the design change separately. It now confirms the cancel target by matching order date, name, and direction; skips cancellation entirely near the close; and avoids prematurely marking an uncertain order as "confirmed rejected."

The discipline that separates "it doesn't work" from "can't tell"

The back half of the week shifted from execution to "how do you reliably tell whether something is an improvement at all."

Thursday, I re-examined the claim that "the prediction score is low." The model learns and is scored over a one-day horizon, but actual trading buys the next morning and holds for several days.

Rescoring the same model over the actual trading horizon gave a score well above zero, not near it. The prediction power wasn't absent — trading costs ate a large part of it, and what remained leaned toward flagging names to avoid.

So the roadmap wasn't "add data first." It was: align the model's training target to how I actually trade, and only then test public data one source at a time.

To test that first candidate, I nailed the pass bar, windows, and seeds into a preregistration document before seeing a single number. Once you've seen the numbers, resisting the urge to nudge the bar toward "pass" is hard.

The verdict has only three values: pass / reject / indeterminate. And "indeterminate" means "keep current," not "let's run it a bit more."

The same discipline applied to other experiments. Six variants narrowing the traded universe, run over a six-year span, all came back indeterminate, so the universe stayed as is.

I also put the same yardstick on retraining deployment. A newly trained model can only ship once it's confirmed no worse than a fixed anchor model, and it's designed fail-closed — if that can't be confirmed, the deploy is blocked.

Sunday, I ran and read out the training-target redesign experiment per the rules locked the day before. The result was zero passes, so current stayed.

Here came an important correction. The natural reading — "ranking improved but it didn't translate to money" — was overturned by another AI right after the readout.

The key was statistical power. The smallest effect this experiment could detect on the return side was larger than the strategy's entire edge.

So the right statement is not "it has no effect" but "whether it has an effect can't be measured from this data." Split apart, the cost savings from lower turnover were clearly positive, while the gross-return change sat in the noise.

As follow-up, I decided to carry the power actually measured this time into the preregistration design of the next candidates. The point is to design experiments already knowing the effect size they can detect.

Why it matters

In one line, this was a week spent asking whether the differences I was seeing were real.

Without fixing the measurement, I'd have mistaken a down-market loss for failed stock selection and kept whittling the wrong signal. Only after fixing the benchmark did the real question — and the real decision — come into view.

Redefining the goal as absolute return pulled the direction of improvement to one side. The question shifted from "how do I beat the index more" to "how do I cut costs and lose less."

All the execution-layer work pointed that way. But over two days I relearned that sound logic still produces wrong results on wrong input (a corrupted previous-day close) or in an under-isolated environment (tests firing real notifications).

The hardest habit to internalize is separating "it doesn't work" from "can't tell." Both end in "keep current," but the first closes a door while the second hasn't even opened one yet.

The only way to resist the temptation was one thing: freeze the pass bar before seeing the numbers, and leave what can't be detected as "don't know."

#quant #trading bot #devlog #operations