Audit findings get measured before they get fixed
A full bug audit of the live EA produced 20 findings. Among them, two 'deviations from the verified code' that did not get reflexively fixed but measured first: one fix would have been meaningless, one was small but real. How not to be fooled by a single tester window, and the philosophy of keeping research and production in lockstep.
At one point I put the roughly 2,500 lines of my live EA’s source through a full bug audit. It produced 20 findings, ranging from genuinely dangerous holes (a halt latch that evaporated on restart) to a subtler species: places where the EA’s behavior quietly differs from the verified research code.
The dangerous ones were fixed in priority order (persistent halt state, an un-stopped-position loophole, order price normalization). This article is about what came after: two “deviation from research code” findings. The normal instinct says research and production must match, fix everything. This project’s rule says something else: measure before you fix. Fixes carry their own risk and cost, and as you will see, the value of fixing turned out to be completely different for the two cases.
First time here? The premise
For new readers: this blog is a verification diary. I (one person) build my own automated FX trading program (an EA, a strategy running on MetaTrader 5) and publish every test, wins and losses alike. “Study N” refers to my numbered research log.
My setup is slightly unusual: there are two codebases. A Python research stack measures methods on 11 years of real data; the MT5 EA is a port of whatever survives. Two terms suffice for this article: PF = gross profit over gross loss (above 1 is profitable), DD = how far equity falls below its peak.
What the research side says the system should look like. The audit in this article exists to keep the live machine from drifting away from this curve.
Why “deviation” does not mean “fix it now”
Some context. In my workflow, the Python research codebase is the source of truth and the MT5 EA is its port. Porting involves judgment calls: which sleeves get which filter, how far an overlay reaches. When a porting-time interpretation diverges from the research side, the thing you verified and the thing actually trading drift apart.
Divergences come in two kinds: those that change performance and those that do not. The first kind deserves fixing. Reflexively “fixing” the second kind means taking on a fresh behavior-change risk for zero benefit. And there is exactly one way to tell them apart: run the backtest both ways and measure the difference.
Case 1 (audit item D3): the fix that would have meant nothing
The first case concerned the equity filter’s reach. This defense halves position sizes when US stocks fall below their 200-day line. The research code applies it to the trend core only; the EA also applied it to the index sleeve and the second satellite. The filter is active on 25% of all days, so three sleeves trading at different sizes for a quarter of history hardly sounds ignorable.
Measured over the full history plus period splits:
| Version | PF | Monthly | Max DD |
|---|---|---|---|
| Corrected (matches research) | 1.69 | +1.060% | -8.33% |
| EA as-is | 1.68 | +1.039% | -7.98% |
A 0.021% monthly gap, with the as-is EA actually 0.35 points shallower on drawdown. Out of sample (2023 onward) it is +1.392% versus +1.405%, a coin flip. Trade count identical at 8,478 (the filter only scales sizes).
Verdict: no difference. There is no performance case for changing live behavior. And the real yield was the byproduct: for the first time, numerical proof that the live EA’s results (+1.039% monthly, -7.98% DD) barely deviate from the published research values (+1.060%, -8.33%). I went to investigate an audit finding and came back with a health certificate for the whole system.
Case 2 (audit item B3): the fix that was small but real
The second case was an overlay on the calendar sleeve. The EA applied a volatility-scaling-plus-equity-filter overlay there that the research code never had, an innocent porting-time “treat it like the other sleeves” decision.
Restoring the research-faithful behavior produced a scene: the MT5 strategy tester (USDJPY, 2022 Q1 window) jumped from PF 1.27 to 1.34. An improvement of 0.07 is big. Too big. Exactly the moment for the full-history discipline.
| Version | PF | Monthly | Max DD |
|---|---|---|---|
| After fix (matches research) | 1.69 | +1.060% | -8.33% |
| Before fix | 1.67 | +1.049% | -8.72% |
The true difference: PF +0.02, monthly +0.010%, drawdown 0.39 points shallower. The tester’s flashy +0.07 came from one three-month window that happened to exaggerate the gap.
But the direction was genuine. Split the history into halves and both halves improve by about the same amount in the same direction (PF +0.02, monthly around +0.01%, DD shallower). A lucky window would have flipped signs somewhere. Small, robust, real, and principled besides: the fix restores verified behavior rather than inventing new behavior.
What the contrast teaches
On the surface, both cases were “research versus production deviations”, both were fix candidates. Only measurement separated the meaningless from the correct. Three rules distilled:
- Measure audit findings before fixing them. Money-leaking holes get fixed immediately; mere mismatches are behavior-change risks and get measured first
- Never trust the size of a single tester window. Borrow its direction, judge on full history plus period splits. A window will happily exaggerate a difference threefold (0.07 versus the true 0.02)
- An undocumented deviation from verified code is a liability even at zero performance difference. Case 2 shows deviations can quietly sit on the losing side, and enough of them make “what exactly did we verify?” unanswerable
The scariest failure mode in running an EA for years is not a big bug; big bugs announce themselves. It is the slow, silent drift between what you verified and what actually trades. These two cases double as an audit report that the drift-counting machinery works. Fittingly, the EA was later refactored into 13 modules with machine-verified behavioral identity (2,009 lines of normalized code matched exactly). Measure first, then move, applies to code structure too.