Comparing GPU Performance After a Hardware Swap - Why the Verdict Flipped Twice

My first direction call was wrong, and my later purchase call was wrong too, for two completely different reasons — the root cause both times was how I was reading benchmarks

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

I swapped my GPU for one from a different brand, and had to pick which execution mode to run it in.

There were two candidate modes, and I judged one clearly better at first. That call turned out to be backwards, and the verdict flipped again after that. Writing this down to work out why it kept happening.

First verdict — the direction was backwards

Moving to the new card meant choosing between two execution modes. I looked at some external reference numbers to compare them, and concluded that the speedup technique I was already relying on (predicting several tokens ahead instead of one at a time) simply didn't work well on one of the two modes.

If that conclusion held, dropping the technique was the right call. But looking again the next day, it was the opposite. The technique wasn't unsupported at all — it produced garbage output, which is a correctness problem, not a performance one.

The root cause was a misread reference number. One of the two figures I'd cited turned out to belong to a different kind of model (one with far less active computation per token), and the naming was close enough that I mistook it for the model I was actually running. Working backwards through the math showed that number was physically impossible for the model structure I was using — that's what exposed the misread.

Once the direction was corrected, I went ahead and deployed. A single card ended up matching or beating the throughput of the two older cards it replaced, and I tuned batch settings on top and confirmed the numbers directly. At that point the verdict looked settled.

Second verdict — flipped again while evaluating an additional purchase

A few days later, I had to re-evaluate whether adding another card, or a different card entirely, was worth it. This time I gathered several external benchmark numbers into a comparison table.

Going back over the table, I found one row where the evidence column was completely empty. Three of the four core arguments in the source material that row cited had already been contradicted by our own measurements.

A deeper problem surfaced too. I'd been converting throughput differences between cards using a memory-bandwidth ratio, but that conversion doesn't hold once the speedup technique is turned on. With it on, next-token computation stops being purely bandwidth-bound in places, and the ratio formula didn't account for that — it ended up underestimating one card and overestimating the other, wrong in both directions.

Cross-checking against newly found external measurements the same day turned up yet another flaw. Most of the external benchmarks I'd been citing hadn't controlled for input context length at all. Throughput drops sharply as context grows, and I'd been applying numbers that ignored that to a workload that actually runs on much longer context.

Third check — a second review caught still more

I thought that was thorough enough, but having a different AI review the same material turned up three more corrections.

One was a model-structure mismatch. A figure I'd used to compare the currently running model against a candidate card actually came from a much heavier model structure. The conclusion (hold off on buying) didn't change, but the stated reason for it had to be rewritten.

Another was about the direction of my own correction. For non-speculative comparisons — with the speedup technique off — the bandwidth ratio formula still held; I'd overcorrected by declaring it invalid across the board. It was only invalid for the regime where that technique is on.

The third was an argument that collapsed entirely. I'd blamed a language-mixing glitch on quantization precision, but a different model running at the exact same precision showed none of it. The counter-evidence had been sitting in hand the whole time.

How I finally settled it — a cheap direct experiment

Every extra round of review meant reading external benchmarks more carefully, and every round turned up a new flaw. So I changed approach.

I returned the expensive extra card I'd already ordered, and instead planned a low-cost direct experiment on the hardware I already had — actually measuring what a one-step precision upgrade buys, within a few days, instead of arguing from a catalog.

No matter how carefully I cross-referenced external benchmarks, none of them matched our actual working conditions (context length, model structure, whether the speedup technique was on) exactly. A direct experiment always does, by construction.

The general lesson

Looking back, the two mistakes looked similar on the surface but had different roots. The first was a simple misattribution — citing the wrong source for the wrong model. The second was citing the right source but applying it under conditions it was never measured under.

After both, I added a few checks to how I evaluate hardware or product comparisons. First: does the number I'm citing actually come from the same conditions I'm comparing against — same model structure, same auxiliary techniques enabled, same workload?

Second, whenever I use a conversion formula, I check what assumptions it depends on. One condition changing — in this case, whether a speedup technique is on — was enough to break the formula, and I didn't see that until it already had.

And now, if a verdict flips more than once in a short window, I treat that as a signal that it needs a third, independent review rather than a fourth pass at the same sources. When catalog comparisons stop converging, a cheap direct measurement gets to the answer faster than reading benchmarks more carefully.