Which Makes More Money, the Free AI or the Paid One - a Simulator That Landed on "Can't Be Measured"

This is the English version of a post originally written in Korean. See the Korean original and other posts(new tab).

There are two candidates for the "brain" that produces the quant bot's stock recommendations.

One is a free model I run directly on my own machine. The other is a much larger model, but it costs money every time I use an external paid API.

The hardware running the free model is a consumer gaming GPU. One card (16GB) comfortably runs a lightweight open-source model — in practical terms, roughly on par with the old free tier of ChatGPT (GPT-3.5).

Adding a second identical card and splitting the workload across both lets me run a much larger model. It's hard to draw an exact equivalence, but in practice it lands somewhere close to the lightweight free tiers each company currently offers (things like GPT-4o mini or Gemini Flash).

By contrast, today's top-end card (the 5090, priced around 7 million KRW alone) can run a similarly sized model much faster, with no need to split anything across cards. My two-card setup is a much cheaper compromise that reaches a similar capacity range.

A natural question follows — which one should I actually run live trading on to make more money? I built a separate simulator to try to measure that properly.

What it takes to compare them fairly

The easiest mistake when comparing two models is running them under slightly different conditions and then blaming any difference in outcome on the model.

So I built this simulator on a "paired comparison" principle. Same dates, same inputs, same trading rules — I run two portfolios side by side, changing only the model.

That way, any final difference in returns can, in theory, be attributed entirely to the model.

But I hit a wall

Once I actually ran it, there was an unexpected wall. I couldn't even tell whether there was a return difference between the two models in the first place.

The reason is a bit deflating. Stocks can only be bought in whole shares, and when a portfolio has few positions, the noise from "do I buy one more share or not" is far larger than any real skill difference between the two models.

Add ordinary market volatility on top of that, and the signal (the model difference) gets completely buried in noise.

Buried how badly? The math worked out to needing tens of thousands of trading days to statistically distinguish this difference with confidence. Even collecting data for years wouldn't get close.

So this isn't "the two models are the same." It's a different kind of conclusion: "the amount of data I could ever collect in my lifetime isn't enough to measure the difference."

If you can't grow the sample, change how you measure

I could have just given up here, but I changed my approach instead.

Instead of collecting more data, I tried measuring the same question multiple different ways. If the different methods converge on the same answer from different angles, that's trustworthy; if they diverge, that divergence itself becomes a diagnosis.

I also built a ruler. I measured the return volatility that comes from picking stocks completely at random with no information, and used that as a "zero baseline." If the model difference is smaller than this random noise, it's effectively nothing.

I ran several independent methods this way. All three landed in the same place.

Model choice turned out to be one of the smallest levers among everything that drives returns — barely distinguishable from picking randomly.

Even the benchmark was a trap

I learned one more thing while measuring: picking the wrong thing to compare against skews the answer before you even start measuring.

At first I planned to use the market index as the win/loss benchmark. But our market index is dominated by a handful of mega-cap stocks, more than half the weight.

Using that as a benchmark creates an illusion — you end up measuring "how did those few stocks do that day" instead of "did my strategy do well." So I switched the benchmark to a hypothetical portfolio holding every stock at an equal weight.

What actually decided the outcome

So what actually drove the results the most? Surprisingly, it wasn't the model — it was the safeguards.

During the simulation period there was a stretch where the market crashed hard, and the amount of loss the circuit-breaker/halt mechanism (the kind that stops trading once a price drops past a certain threshold) prevented was several times larger than the return difference between the two models.

I'd started out thinking this was a problem of picking a smarter brain. What actually protected the money turned out not to be the brain, but the "reflex to stop when things get dangerous."

So where is the money, then

At this point I had to reframe the question itself — what actually needs to happen to make more.

Working it out, the return you actually end up capturing breaks down roughly as: the return you could theoretically expect, multiplied by how much of it you actually captured (the capture rate), minus the money that leaks along the way and the money lost to accidents.

What matters here is that the first term — "expected return" — isn't something I can force upward. The "can't be measured" finding above already showed that picking a smarter model doesn't raise that ceiling.

So the only things I can actually influence are the rest: capturing the expected return that's already there without letting it leak, and not losing big during a crash.

Reducing "leaks" — idle cash, trading costs, orders slipping away from the intended price — and using the safeguards above to prevent big losses. That's where the money actually was, not in a better brain.

So I moved the decision axis

If returns can't distinguish the two models, then the basis for choosing between them needs to move somewhere else entirely.

If returns are effectively equal, what's left is operational reality — free versus paid, fast versus slow response, how well it tolerates incidents, and whether it can be reproduced identically later.

This is a question that's largely already answered even without simulation. Running for free, being unaffected by external outages, and being able to rerun identically at any time — all favor one side.

The simulator's role changed too

Originally this simulator was supposed to become a scoreboard deciding "which model won."

But once it became clear that score itself wasn't measurable, its role changed. Now it's closer to a disaster detector.

Its job is to catch execution-stage failures before they happen with real money — cash math not adding up when actually running a portfolio, failing to stop when it should have stopped, accidentally buying the same thing twice. Catching these traps before putting real money in turned out to be a far more valuable output than an unmeasurable return number.

The lesson I took from this — "watch for where money leaks" — carried over into small gauges that now run quietly every day. They place no real orders at all; they just shadow-measure and log whether money is leaking or returns are being missed.

Where this simulator sits

This tool sits, within the overall quant bot architecture(new tab), at the stage right before real money goes in.

Fill prices at trade time are reproduced from actually collected minute-level market data. This price data is pulled from a separate data archive project(new tab).

The code wasn't split off into a separate repository — it lives inside the main project. Since all its inputs already live in the main project, splitting it out would just add friction pulling things from here and there. For a solo project, over-separation isn't tidiness — it's debt.

"Can't be measured" is also a conclusion

Looking back, the real payoff of this project wasn't some flashy return number — it was honestly confirming that this question can't be answered this way.

At first it felt deflating. All that effort building a simulator, and it couldn't even measure the thing I originally wanted to know.

But thinking it over, if I hadn't accepted "can't be measured" and had instead forced out some number and claimed "this model won," that would have been more dangerous — mistaking noise for skill.

Sometimes the answer isn't "collect more data and you'll know." Sometimes the answer is "this is a kind of difference you can't measure at all." And that answer redirected the time I would have spent swapping brains for no reason toward where the real money actually was — plugging leaks, and not losing big. Accepting that is a conclusion too, and that's what I learned this time.