Devlog
All Devlog posts (116)
- Re-Investigating the Flip-Flop Alert - The Cause Was a Change in Stock Count, Not Band OscillationAn alert fired again because the trading system bought a stock and sold part of it right back the next round - a "flip-flop." The design had blamed price jitter at the rounding edge, but re-tallying two months of execution logs read-only showed the real cause was the number of target holdings growing, which shrinks each stock's target amount. Here is how I decided to keep observing instead of flipping the switch off.
- The Morning Round Misfired on a Late-Open Day - Gating It on the Real Open TimeOn a day the market opens an hour late, restarting a stopped schedule timer instantly fired its already-elapsed trigger, so the morning round sent orders into a closed market. Every order was rejected, and because that round had claimed the execution slot, the later rounds were blocked too. This is how I tracked it down and made the system read the actual open time from the calendar so a pre-open round holds itself back.
- The First Live Order Cancel - A Broker Rejection and an Order-Number Type MismatchThe path that cancels an unfilled sell order mid-session and re-plans fired on a real account for the first time, and the broker's cancel API rejected it over an order-number type mismatch — with the rejection reason missing from the log. This is how I tracked that down, normalized the order number to an integer, and made the rejection reason print — plus how an external AI advisory checked the safety brake that blocked every buy on a rebound morning.
- Price-Based Trading Rules All Failed the Backtest - Switching to an Order-Free Forward RecorderI spent a whole day backtesting whether the price fields in my recommendation reports (entry, stop, target) could squeeze out more return, and every rule I judged fell short of the bar. So instead of putting any of them into live trading, I stood up an order-free forward recorder to judge them months from now. This is that process, plus a pre-registration rule I caught myself breaking.
- First Live Trading Day - Most of the Safety Machinery Never FiredThe first time I ran the live-order code on a real trading session, orders filled cleanly and none of the safety paths — margin retry, order cancellation — ever triggered. This is about how I decide to trust safety code that has never fired, and how I blocked a calendar edge case the code didn't know about (a special late-open exam day).
- Fixing the Measurement, Then Redefining the Goal - Absolute Return and Execution-Layer CleanupClearing away a benchmark illusion showed the stock-selection edge was essentially zero, the owner locked the system's goal to roughly 10% absolute return per year, and the "always fully invested" principle was scrapped. The rest of the week went to the intraday filter, order retries, and measurement discipline that decision demanded.
- Reading Out the Label-Redesign Experiment - Zero Passes, Not "No Effect" but "Can't Measure"Yesterday I froze the rules first; today the results of the label-redesign experiment came in. Zero passes, so I kept the current setup, and an outside AI review corrected my reading from "it doesn't make money" to "the data can't tell whether it makes money."
- A Roadmap for Improving Predictive Power, and the "If You Can't Tell, Keep Current" RuleRe-scoring the same model on the window we actually trade showed its predictive power wasn't as near-zero as it looked. Six years of narrowing-the-universe experiments all came back "can't tell," so I left the universe alone, and I added a non-inferiority gate against a fixed anchor model for retraining deploys.
- Retrying Margin-Rejected Buys and Cancel-Replanning Stale Orders on the Live AccountSome first-round buys on the live account were rejected for insufficient buying power, so I added a guard that retries the order once in the same round if the cash shows up. Orders that sat unfilled all day because of a stale planned price now get canceled and re-planned just before the next round.
- Intraday Trade Filter Fixed to Use the Regular-Session Close - A Portal's Daily-Bar Close Definition Had ChangedThe intraday filter I added to the live account yesterday was judging spikes and drops against a wrong reference (the prior-day close). A portal's daily-bar "close" had started including after-hours trades, so I recut it to use the broker API's regular-session close as v1.1. The same day, a regression run accidentally fired six real notifications.
- Setting the Trading System's Goal - Absolute Return ~10%/yr, Scrapping the Always-Fully-Invested RuleYesterday I wrote that "stay positive even when the index drops" and "never leave cash idle" couldn't both be true, and that choosing between them was my call. Today the owner decided: this is a profit system aiming for a long-run absolute return of around 10% a year net of costs, and the rule that it must always be fully invested is gone. As a first step, an intraday filter that briefly defers trades on sharp spikes and drops went live on the real account.
- Redefining the Performance Benchmark - Replacing a Period-Mismatch Illusion With a Stock-Selection MetricMy live-account report looked like it was badly trailing the index, but that was an artifact of measuring the numerator and denominator over different periods. After redefining the benchmark as a per-lot like-for-like comparison, an honest measurement surfaced a more uncomfortable question — the stock-selection edge is essentially zero.
- Token Ping-Pong and an Overnight Batch Hang - Two Root-Cause HuntsTracked down two operational incidents this week - an auth token that kept invalidating itself, and a nightly batch job that silently hung overnight - and set up automated checks to confirm the fixes actually held.
- Overnight Scoring Batch Stall - Two Unbounded External CallsAn overnight stock-scoring batch stalled; fixed the first timeout-less call, then asked whether the fix was enough and found a second one hiding in the same codebase.
- Toss Token 401 Ping-Pong - The Real Root Cause Was a Missing Cache Re-checkFinally tracked down why an external quote API's auth token kept invalidating itself, fixed the re-issue logic, and set up an automated check instead of relying on memory to confirm the fix held.
- From an Exchange Lockout to a GPU Teardown — a Week of Chasing Root Causes All the WayI downgraded an exchange-lockout crash into a quiet failure and finished a one-share live-trading test by hand. A recurring token-invalidation bug turned out to be caused by restart-order mismatch, and a secondary GPU cleanup went all the way through instead of stopping halfway.
- Removing a Secondary GPU - A Fail-Loud Registry Flagged the Blast Radius InstantlyPulled a secondary graphics card ahead of a planned sale, and the fail-loud checks I'd already built pinpointed the exact impact the moment the system rebooted.
- Recurring Toss Token Revocations - A Restart-Order MismatchTracked down why an external quote API's auth token kept invalidating itself, and decided to wait for the final verdict on a new-exchange entry experiment despite failing one criterion early.
- Confirming a One-Share Live Test Path - Manual Orders Over AutomationPlanning a one-share buy-and-sell test on a live account, I chose manual order entry over reviving a retired automated path. Also rolled up a batch of changes and restarted two resident processes today.
- Merging a Looser Exit Rule - Sorting Real Failures from Pre-Existing Ones After a RestartA looser exit-timing rule and a round of paper-broker fixes landed on the same day, followed by a resident restart. The full test suite then turned up some failures, and each one had to be checked against the merge before trusting it.
- KRX Access Blocked - Demoting an Import-Time Crash to a Login FailureAn exchange IP block turned a data library's import step into a process-killing crash, and the same day also saw a new research hypothesis go from preregistration to merge.
- Re-judging Performance, Going Live on Early Entry, Two Rounds of Review on a Fund-Transfer Feature — a Week of Double-Checking NumbersI had a strict AI-led re-judgment of performance against the benchmark, pushed an early-entry structure live, and only caught a silent reporting bug and hidden money-safety flaws by checking things myself a second time.
- Cross-Account Fund Transfer - Code Review Caught Two Ways Money Could Go MissingBuilt a fund transfer feature ahead of a planned account restructuring in mid-December, and an independent review caught two separate ways money could silently vanish or get fabricated.
- Weekly Blog Report Skipped for Five Weeks Straight - A Dead Account Reference Was the CauseTracing why the weekly performance report kept silently skipping turned up a reconciliation check still reading state from an account the system had already migrated away from.
- First Live Day of Early Entry, and a Two-Tier Structure Stalled on FundingEarly entry — freezing the ranking ahead of time and pulling the first trading round forward — went live, while yesterday's proposed two-tier structure stalled for lack of capital to fund it.
- Rechecking Live Performance — A Cold-Blooded Verdict Against the Index, and a Design That Actually Makes MoneyAsked an AI advisor for a hard-nosed verdict on whether live returns are beating the index, then pushed it further for a structure that actually pays off after costs.
- Consolidating Broker Paths — Swapping Out the Paper-Trading Comparison TrackRetired an old paper-trading comparison track for a new simulated-broker one in a single day, while deliberately leaving a way back.
- Experiment Bugs, a GPU Incident, and a Live-Account Buy Block — Turning Manual Intervention into Something MonitoredThree incidents this week each got handled by hand — then followed by work to make sure the same intervention would next time be logged and alerted on automatically, or not need a human at all.
- Ending a 60-Day Shadow Experiment, By the Rule I Set in AdvanceAn experiment that didn't clear its pre-registered statistical bar got retired on schedule, and a data correction plus an old incident's follow-up got closed out the same day.
- A Real-Account Manual Intervention, Patched With Eight Safeguards in One DayA buy block on a live account left cash piling up, so I traded manually through the broker app — and closed every gap that surfaced along the way, same day.
- A GPU Driver Mismatch, Resolved by a Reboot With Nobody at the KeyboardAn automatic update left the kernel module and the userspace library on different versions — recovery, from reboot to verification, ran unattended.
- Running a Prompt A/B Test, I Hit Three Bugs Back to BackThe experiment itself was supposed to decide whether a new prompt line helps — instead, the harness running it broke three different ways in one evening.
- Extending the Order-Execution Safety Layer, Building a Pre-Registered A/B FrameworkThe execution layer got touched almost daily — output caps, reconciliation guards, partial-fill buys, deposit/withdrawal handling — while the real discipline was building a framework that locks in criteria before seeing results, instead of shipping on instinct.
- Pulling the Deposit/Withdrawal Money-Path Forward and Shipping It Same DayI'd parked the deposit/withdrawal bookkeeping (the "money path") for a weekend window, but at the owner's call I pulled it into today — and the critical bug the code review caught lived not in the logic but in a failure-message.
- Partial-Fill Buys, and Pre-Registering an A/B Prompt ExperimentI changed the system so that an order short on cash gets scaled down instead of rejected outright, and instead of rushing two half-baked ideas into production I set up a proper pre-registered A/B experiment.
- Fixing a Stock Classifier Misclassification and Correcting a Paper-Account Reconciliation GapI found and fixed a bug that mistook actively-traded stocks for halted ones and kept blocking buys on them, then traced and corrected a reconciliation gap in a paper trading account.
- Shipping the Output Token Cap and Expanding Order-Ledger Reconciliation GuardsI rolled out yesterday's output-token-cap fix to both serving paths, and added two more layers to the order-ledger reconciliation safeguards.
- Backup GPU Self-Healing Collapses, Paper-Trading Fill Mismatch TracedBackup GPU self-healing collapsed completely on its first night in production and got redesigned, and a paper-trading ledger residual turned out to trace back to a fill mismatch from nine days earlier.
- A Concurrency-Scaling Smoke Test, and Discovering There Was No Output Token CapI tried to raise the serving concurrency for more throughput and the smoke tests caught it, and along the way I found out the output side had no token cap at all.
- Expanding Live Trading to a Shared Account, Doubling Up on GPUs, and the Measurement Tools That Kept LyingThis month's progress was expanding live trading to a shared account and doubling up on GPU capacity, while I kept finding defects in the measurement tools themselves — benchmarks, reports, safety devices
- Tracing a Paper-Account Reconciliation Gap, and Retiring a Dead AlertA fill from a completely different stock, placed nine days earlier, had gotten glued onto the wrong sell order — and I also retired a registry entry that had been alarming pointlessly for over a month.
- R9700 Backup GPU Total Failure - Getting the Cause Wrong TwiceThe self-healing I wired in last night didn't work at all this time, and while bringing back missing stocks I reversed my diagnosis twice.
- R9700 Backup GPU - Wiring Self-Healing In and Rechecking the Concurrency BumpI wired an actual fix action onto yesterday's dead-slot detector, and found that raising concurrent requests further was based on a wrong premise.
- Stacking Safety Nets While Re-Checking Whether Fixes Actually WorkedFrom a server degeneration guard to a dead-slot detector on GPU, several safety layers went up this week — and repeatedly, re-checking whether a fix actually achieved its goal turned up a wrong premise underneath.
- R9700 Backup GPU - Bottleneck Root Cause and a Dead-Slot DetectorI found why the backup path doesn't get faster with more concurrent requests, and built a detector that catches a slot dying and spewing runaway output.
- Qwen3.6-35B-A3B on an AMD R9700 with llama.cpp: Vulkan vs HIP, Concurrent Slots, and MTP Measured in tokens/sVulkan beat HIP, four concurrent slots was the knee, and MTP only helped with a single request
- Retiring self-consistency checks and catching a test that was polluting the live ledgerI fully turned off a measurement I decided was wasting resources, and in the process found a test suite that had been writing fake rows into the live trading ledger.
- Qwen3.8-27B on a Single RTX 5090 with vLLM: NVFP4 + MTP Tuning Results and the KV Cache BottleneckMeasured tokens/s while going from 3 to 14 concurrent slots to MTP + 8 slots: how I cut a 100-stock nightly batch from six-plus hours to under three, and why you shouldn't fully trust that number
- Making News Collection Real-Time and Shipping a Standby-GPU SafeguardA structural change to news collection solved what a simple time shift couldn't, and a safeguard for running the standby GPU full-time shipped to production today.
- Shifting a Nightly Run Time and Reviewing a Standby-GPU DesignVerification showed the time shift alone didn't achieve its original goal, and a design review for running the standby GPU full-time turned up a misread performance figure from the past.
- Closing Out an Order-Identification Gap and Putting the Task Doc on a DietI patched a leftover hole in the execution layer, and reined in a task-tracking document that had been growing out of control.
- The Server Was Quietly Degrading - Building a Detect-Isolate-Recover SafeguardRight after closing out yesterday's news-input incident, a new defect showed up, and I built a safeguard against it in a single day.
- I Finally Changed the Temperature for Real - and Overrode a Pre-Registered Rule That Said "Reject"The sampling-temperature experiment I believed I'd already run for weeks had never actually been executed
- A Week of Catching Measurement Tools Lying to UsA universe-contamination incident kicked off a week where ranking logic, AI sampling settings, and a new benchmark tool each turned out to be quietly wrong
- Temperature Experiment — Direction Was Clear, Size Wasn't, Decision DeferredThe two-day comparison test came back with an ambiguous verdict, so I had another AI independently re-check it, and cleared a backlog of fixes on the benchmark harness while I waited.
- Benchmark Harness — Auto-Resume Hit a Permission Wall and Wasted GPU TimeThe corrected tool from yesterday resumed automatically overnight, but it launched one trial and quietly froze. I ended up taking over by hand to finish the round.
- Benchmark Harness — Two Measurement Bugs in the Measuring Tool ItselfThe tool was overcounting GPU usage by more than 3x, and a too-short measurement timeout was flipping candidate rankings upside down. Found and fixed both in one day.
- First Run of a Benchmark Harness — Plugging the Leaks One by OneI found that yesterday's model setting fix hadn't actually reached the new benchmark tool at all, spent the day plugging leak after leak, and also finally closed out a set of safety items that had sat untouched for 11 days.
- Checking the AI Judgment Models' Sampling Settings — Wrong Values All AlongThe local models behind my AI reports had been running with the wrong sampling settings for weeks, and I also closed out last week's graphics-card incident.
- Replacing the Tie-Break Logic — from a Rejected Method to a New OneA tie-break method for ranking failed a pre-registered evaluation bar, and I swapped it for a new one the same day.
- The Day Market-Cap Data Came Back Empty - A Universe Contamination IncidentWhen market-cap data came back entirely empty, stock rankings silently reverted to alphabetical order — so I added three layers of defense to catch it next time.
- Moving to a Shared Live Account - The Same Missing-Account-ID Bug Kept RecurringI started live trading through a new shared account, and along the way the same class of bug kept resurfacing - code that forgot to pass or accidentally shared an account identifier, triggering both a false kill-switch trip and a wrongful forced sell
- Kill-Switch Postmortem - the Root Cause Was Still ThereYesterday I patched the symptom of a false kill-switch trip in a hurry. Today's postmortem found the root cause was still live and could have blocked Monday's real trading again.
- A False Kill-Switch Trip From a Missing Account TagAn information-only simulation tick shared the same kill-switch namespace as the live account, and a fake loss computed from an unrelated account's balance nearly froze Monday's live trading.
- A Do-Not-Rebuy Flag That Force-Sold the Wrong Account's HoldingsA flag meant to stop one account from rebuying transferred-out stock turned out to be global, and it force-sold the same stock right after another account received it. Also finished building a new automation tool for performance tuning.
- Two Bugs on Account-Migration Day, Plus the Real Root Cause of the GPU FailureThe physical share transfer went through, but the cash auto-sweep sat broken all day and a live account hit a reconciliation incident. Separately, the recurring GPU-detection failure finally got a real fix.
- Redesigning a Brokerage Account for Shared Use — and the First Live OrderA new brokerage account mixes personal and bot funds, so I rebuilt the safety layer around that, sent the first live order through it, and finalized a bank-transfer-free way to move funds between accounts
- Closing a Login Bug That Could Have Leaked Broker CredentialsTracking down why brokerage API login kept failing led to a near-miss information leak that never actually shipped, plus a new buffer rule for how stocks enter and leave the trading universe
- A VRAM Cleanup Patch Came Back as an Evening Production OutageA GPU memory cleanup applied earlier in the day triggered three straight production failures that evening, and I also fixed a brokerage API pagination bug
- Tracing Recurring Ledger Corruption and Automation Traps Back to Their RootsI finally found the root cause of ledger corruption that recurred two days in a row, and opened new experiment tracks with an intraday observation pipeline and a GPU expansion along the way
- Chasing a 100-Stocks-in-10-Hours Goal for a Local LLMTracked down why a setting that won on benchmark lost in production, and closed out a statistical alert I'd been re-litigating for days
- I Finally Bought a Card I'd Rejected Several Times - AMD Radeon AI PRO R9700The story of buying a card I'd turned down as a main GPU more than once, for a completely different purpose
- Closing Out Yesterday's Workaround and Two Misfiring BugsFixed the hardcoding that blocked manual trading rounds at the root, and tracked down why a ledger-contamination bug recurred and why an overnight recovery guard kept misfiring
- One Overnight Crash, One Full Day of CleanupA wiring mistake broke the overnight batch; recovering from it also exposed a bypass loophole and a two-day-old test-contamination bug
- A Live-Market Observation Pipeline Goes Live, and a Candidate-Pool RebalanceThree bugs surfaced back to back the moment a new pipeline first fired, and the same day I rebalanced which stocks even get considered
- Four Stages of Refactoring, a Follow-Up Review, and Automated RestartsI asked an AI to review the whole project's structure, and once the fixes were in, I asked it to check its own work
- Going Live and Onboarding a Second Broker, With Quiet Bugs Along the WayThis month's biggest progress was starting live trading and running a second broker in parallel, while I kept turning up bugs that had been failing quietly for days
- A Missed Process Restart and a Paper-Account Ledger Corruption, on the Same DayA ranking process that silently sat idle for days, and a brokerage API field that quietly shrank order prices by a factor of 1000
- Starting a Second Broker's Parallel Run, and the Quiet Bugs Found Along the WayThis week's biggest step forward was verifying a second broker on a real account and starting its paper-trading parallel run, while several bugs that had been failing silently for days got caught along the way
- A Gate That Wouldn't Open for Months, So I Automated the CheckAn automated regime sensor, plus two overnight monitoring gaps a second AI review turned up
- Switching to a New Local Model, and Quietly Keeping the Old One AroundI wrapped up weeks of model comparison and made the switch official, then caught a completion-detection bug in a separate experiment
- GPU Job Time-Window Collisions - Turning Repeated Failures Into InfrastructureI moved the safety-window math I'd been doing by hand into code, and also fixed a live-account trade that got reversed right after it went through
- I Moved This Blog - From Tistory to GitHub Pages and Dev.toTrying to automate the writing process ended up moving the whole platform
- Local LLM Speed by GPU Generation, and Our Production's Daily Token VolumeModel names, serving configs, and full prefill/decode measurements included (reorganized as of 2026-09-26)
- Gigabyte RTX 5090 AORUS MASTER ICE Purchase Review - Consolidating Two GPUs Into OneWhat changed after replacing two split GPUs with a single card
- Re-Sorting a 5,000-Line Backlog Turned Up Bugs That Had Been Failing Silently for DaysGoing through the whole backlog from scratch surfaced two features that had been quietly broken for days, and I started running the second brokerage side by side in paper trading
- A Sequential-Execution Bug from Moving Production Earlier, and Verifying a New Broker ConnectionI fixed a sequential-execution bug uncovered by pulling production earlier, with a batching switch, and verified a new broker's API against a real account
- Tidying Up a Double-Layered Safeguard, and Un-Returning a GPUI collapsed two overlapping defenses into one, and reversed last week's decision to return a new card
- Rebuilding the Publishing Pipeline, Then Catching Signals That Fail SilentlyWhile I was at it changing how this blog gets published, I also worked on judgment that resists misleading samples and instrumentation that catches pipelines dying without a sound
- Weekend A/B Verdict and Execution Pipeline HardeningI made sure not to be fooled by a sample that looked better than it was, and hardened execution instrumentation along the way
- We Swapped GPUs, Then Kept Finding Blind Spots in Our Own ChecksA hardware verdict flipped twice in one week, two verification tools turned out to be quietly missing errors, and a real brokerage account needed two manual human interventions
- Leaving Tistory for GitHub Pages + Dev.to - Overhauling the Publishing PipelineI tracked down the real reason the devlog had gone quiet for weeks, and ended up rebuilding the whole publishing pipeline
- First Week of Live Trading — Prep, Launch, and Follow-Up FixesMonday was one last checkup, Tuesday I started live trading, and over the following two days I fixed problems that only surfaced under real operation
- Weekend Devlog — Reviewing Strategy Hypotheses and a Pre-Live-Trading CheckupThe three weekdays were quiet, but the weekend was packed — I reviewed and shelved three strategy ideas, then a full pre-live-trading checkup turned up bigger gaps than expected
- Wrapping Up Notification Cleanup, a Performance Calc Error, and a RegressionFive days across a weekend — I finished cleaning up the notification system, found that paper-account performance math didn't quite match the real balance, and a day later ran into a small regression from the fix
- US Market Feasibility Check — Technically Possible, Not Yet TimeToday was about confirming that the last few days of fixes actually hold up, plus a serious look at whether this system could move to the US stock market
- A Silent Crash, a Missing Record, and 27 Hours of Signal SilenceThree days spent chasing incidents where "nothing happened" rather than "money went out wrong" — a silent Monday-morning crash, a performance-record gap found via a reader tip, and over a day of signal silence during a circuit-breaker event
- Building a Backup System for the Project's AssetsFound that all the data accumulated since day one was sitting on a single disk with no backup, and rebuilt off-site backup and version control from scratch
- A Cash-Deployment Bug, a Risky-Stock Safety Net, and a Clock-Skew IncidentThree days of tightening up things that already existed rather than building new ones — fixed a paper-account cash-deployment bug, built a risky-stock safety net after a friend flagged a ticker, and cleaned up a clock-skew incident after a reboot
- Caught a Bug I Thought Was a 'Hallucination,' Then Placed the First Real OrderThe last stretch of live-trading prep was too hectic to write day by day, but these two days packed in a data bug and this project's first real trade
- Two Suspicions I'd Been Carrying, Both Turned Out WrongHad another AI advisor stress-test worries about the network-watchdog program — both turned out to be unfounded — and caught one more real safety-layer bug on the live-trading prep side
- Before the First Real Order, I Pulled Out Everything I Couldn't TrustA 15-second WiFi outage set a robustness principle in stone, and auditing the code that actually moves money concluded "don't place the first order yet"
- On the Verge of the First Real Order, the Code Voted NoAuditing the code that actually moves real money turned up several genuine bugs, and it also prompted a fresh look at which broker to use
- Asked in Korean, Got an Answer in English — Then Came Audit SeasonChanging how data was passed in made reports suddenly come back in English, and auditing several parts of the system turned up a real bug and a real blind spot
- The Outage Was 15 Seconds, the Damage Was 4 Hours 45 MinutesWiFi dropped briefly and quietly settled into a half-broken connection, which brought back an old principle — don't make the cheapest thing depend on the riskiest one
- Caught the Same Kind of Bug Again, So This Time I Closed Off the Whole ClassFound another case of a major ticker's name slipping through recognition — this time, instead of just patching it, I restructured things so this class of bug can't come back
- Swapping the Case Unstuck a Backlog of Deferred WorkI finally got the second GPU properly seated, and along the way found that two of the most important tickers had been missing from data for weeks
- Watching AI Be Confidently WrongAn AI reviewing a report flagged real earnings as an "impossible number" and called it data contamination — it turned out to be genuine
- A Week That Started By Doubting Surface Metrics and Ended With Reproducible SpeedMonday's 'cleanest' model turned out on Tuesday to be running on no real grounding at all — and that same distrust of surface metrics and speed gains ran through the rest of the week
- Yesterday's 'Flip', Dug Into Further, Turned Out More ComplicatedSpeed really did improve, but I found out consistency was shaky — and in the middle of it all, a loose power cable knocked the machine into a reboot
- GPU Expansion: Swapped In the New Card First to Test It AloneBefore plugging both cards in together, I checked the new card's stability on its own — and dug up an old idea I'd shelved as "no effect" and flipped it
- Caught Three Things That Were Going Wrong QuietlyFrom a generation that spiraled out of control forever, to a morning fix that quietly halved throughput by evening
- Finally Found the Real Culprit Behind Weeks of Overnight CrashesTracked down and fixed the real cause of a memory problem, and accidentally killed production while doing it
- Trusted a Summarizer's Answer, and an Uncancelled Order Sat on a Live AccountBuilding a track to safely verify a new auto-invest feature, plus an incident from guessing at an API
- One Leg of the Ensemble Had Been Running on Zero Evidence for Six MonthsCatching a research pipeline stage that turned out to be operating with no real grounding, and rebuilding overnight production while I was at it
- Hidden Failure Patterns Found While Comparing Three Local LLMsTwo failure types that grade distributions alone couldn't reveal, plus a structured-output problem fixed by migrating frameworks