When the Simpler Model Won
What a failed financial VAE taught us about representation learning. Across four model families and six trained models, the simplest model reconstructed best, and every layer of added sophistication made it worse.
Across four model families and six trained models, the simplest one reconstructed the data best, and every layer of sophistication we added made it worse.
We set out to build a Transformer Variational Autoencoder, internally called “WaveTek,” to learn compact representations of short market-microstructure sequences. It never turned into a useful trading tool. This piece is about the research itself: what happened when we rebuilt the model correctly, tested it against its own simpler cousins, and let the results override what we expected going in. That’s not really a failure. It’s the result, and it’s worth understanding why.
1. The model I’d built, and the bugs I found
WaveTek was a beta-VAE: an attention encoder compresses a sequence into a
latent distribution, a sample z is drawn, and a decoder rebuilds the
sequence, with a KL term pulling the latent toward a clean Gaussian. On paper
that gives a smooth, samplable space of “possible patterns.”
Reading the code, three problems stood out before we ran anything:
- The latent was disconnected. The decoder reconstructed from the
encoder output, not from the sampled
z. The variational branch produced a latent that nothing downstream depended on, yet the KL term still penalized it. The optimizer’s rational move is to drive the KL to zero and ignore the latent entirely: posterior collapse by construction. - No positional encoding. The model fed a bare feature projection into self-attention. Attention is permutation-equivariant, so with no positional signal a “time-series” model has no sense of time.
- Biased denoising noise. The “Gaussian noise” used the data’s variance as its mean instead of zero: a large positive bias, not a mean-preserving perturbation.
I’d already noticed, empirically, that a plain autoencoder beat the VAE. Now we knew one reason why. But a code review isn’t a result, so we rebuilt it.
2. Rebuilding small, correct, and CPU-only
The corrected model, SmallTransformerVAE, fixes all three:
- Positional encoding added to encoder input and decoder queries.
- A load-bearing latent: the decoder’s only input is
z(via cross-attention), sozhas to carry the information needed to reconstruct. - Zero-mean denoising noise in standardized feature space.
Five unit tests check each fix independently: the decoder provably depends on
z, the encoder is order-sensitive, the noise is zero-mean, and the KL term
is well-formed. The model is deliberately tiny, 45,863 parameters, and
trains on CPU in a few minutes. No GPU, no Lightning, no .cuda() anywhere.
Training data: 17,037 windows of 32 consecutive trades, each with 7 per-trade microstructure features, drawn from Level-1 order-flow sessions.
3. How we tested each model
All six models trained on the same seed-0 data: 17,037 windows of 32 consecutive trades (7 microstructure features each), split 80/20 into train/validation and standardized per-feature on the training split alone. Every model got the same 25-epoch budget, Adam at 1e-3, and zero-mean Gaussian denoising noise (σ = 0.1) added to the input.
But “test” means something different depending on what the model is actually being asked to do, and conflating the three would make the comparison meaningless:
- Reconstruction (Linear/PCA, GRU, Transformer AE, Transformer VAE): encode the full noised 32-step window into a 16-dim bottleneck, decode it back, and score MSE against the clean window. This is the like-for-like comparison behind Sections 5 and 7, where every model sees the whole sequence and has to compress-then-rebuild it.
- Next-step prediction (decoder-only): causal self-attention predicting each timestep from only the real steps before it. There’s no bottleneck, so its number isn’t comparable to the reconstruction column.
- Tail infilling (GLM): trained on random contiguous blank spans, but evaluated specifically by masking the last 8 of the 32 steps and infilling them from bidirectional context over the remaining 24. That’s the same test the original WaveTek used, “hide the end, predict it,” reproduced properly with a real masked-attention model instead of an ad hoc heuristic.
So yes, tail-masking is in here, but only as the GLM evaluation in Section 6. It isn’t how the VAE/AE/linear/GRU reconstruction numbers were produced, and it isn’t how decoder-only was scored either. Keeping those three protocols separate is why Section 8 splits the six models by objective instead of ranking all of them on one shared number.
4. It collapsed anyway
First training run, KL warm-up only:
epoch 1 | active latents 9/16 | val recon 1.000
epoch 5 | active latents 0/16 | val recon 1.046 ← KL crashed to ~0
epoch 25 | active latents 0/16 | val recon 1.047
Reconstruct from true z: 1.047 | from random z: 1.047 (z is irrelevant)We fixed the wiring, and the latent collapsed anyway. As the KL weight ramped up, the optimizer found it cheaper to zero the latent and reconstruct from position alone than to encode noisy order flow and pay the KL cost. Reconstruction sat at 1.0, the trivial “predict the mean” baseline for standardized data.
That’s the first real lesson: posterior collapse is a training dynamic, not just a wiring bug. “Structurally load-bearing” is not the same as “learns to be load-bearing.” A regularizer with no consumer is a tax the model learns to avoid.
The standard remedy is free-bits: reserve a minimum KL per latent dimension so the optimizer can’t zero it out. With free-bits:
epoch 25 | active latents 16/16 | val recon 0.836 | KL 2.55
Reconstruct from true z: 0.788 | from random z: 1.122 (z matters, 1.4x)Now the latent is alive (16/16 dimensions), the decoder genuinely depends on it, and reconstruction beats the mean baseline. A working VAE, modest but real. Order flow does have compressible structure for a latent to hold.
5. The AE still beats the corrected VAE
My original claim, run cleanly for the first time: a deterministic autoencoder twin, identical architecture, exactly matched parameter count (two summed latent heads mirror the VAE’s mu/logvar heads), same data, same recipe. The only difference is the variational bottleneck.
| Model | Params | Val reconstruction MSE |
|---|---|---|
| Mean baseline | n/a | 1.046 |
| VAE (corrected, free-bits) | 45,863 | 0.836 |
| AE (deterministic twin) | 45,863 | 0.747 |
The AE reconstructs about 12% better, for a principled reason this time, not the old bug: the KL term is a constraint, and a constraint can only cost reconstruction. The VAE spends capacity making its latent Gaussian and samplable; the AE spends all of it on fidelity. A VAE only earns its complexity if something downstream needs what the latent buys (generation, smoothness, disentanglement) enough to pay the reconstruction tax. Here, nothing did.
6. Completing the family
Two more architectures round out the original trio, on different objectives (labelled as such):
- Decoder-only (GPT-style, causal): predicts the next step from the real past, no bottleneck. Next-step MSE 0.636, about 40% better than a random walk. It thrives by being order-aware, exploiting the temporal structure the original VAE was blind to.
- GLM (autoregressive blank-infilling): the principled version of WaveTek’s original “hide the tail, predict it” test. Infilling the hidden last 8 of 32 steps from bidirectional context: MSE 0.782, about 21% better than the mean and 40% better than persistence. Naive persistence is actually worse than the mean here: order flow doesn’t carry forward at the feature level, so the model has to learn real context structure.
The decoder-only and GLM aren’t scoring lower because they’re “better.” They keep the real context instead of compressing it, so they’re answering prediction and infilling questions, not representation questions.
7. Swapping model families entirely
We then swapped families completely, holding the objective fixed (16-dim bottleneck reconstruction) so the comparison stays like-for-like:
| Family | Kind | Params | Val MSE | vs baseline |
|---|---|---|---|---|
| Linear / PCA | linear | 7,408 | 0.571 | +45% |
| GRU | recurrence | 46,603 | 0.639 | +39% |
| Transformer AE | attention | 45,863 | 0.747 | +29% |
| Transformer VAE | attn+var | 45,863 | 0.836 | +20% |
A 7,408-parameter linear autoencoder, six times smaller than the others, reconstructs order flow best, and the ranking is roughly the inverse of complexity. The compressible structure at this scale is largely linear; attention and recurrence add capacity for nonlinear structure that mostly isn’t there, and the harder optimization costs more than it returns.
One caveat: this is a matched-budget, matched-recipe comparison (25 epochs, same LR, CPU), not a fully tuned bake-off. Transformers are harder to optimize, and with warm-up, more epochs, and tuning they would probably close some of the gap. So the claim isn’t “linear beats transformers, full stop.” It’s narrower: under equal and modest budget, the simpler family wins by default, and nothing here suggests the added complexity is warranted. In practice, budget and tuning effort are always finite, so this is the comparison that matters most often.
8. The six-model picture
| Objective | Model | Error |
|---|---|---|
| bottleneck reconstruction (like-for-like) | Linear / PCA | 0.571 |
| GRU | 0.639 | |
| Transformer AE | 0.747 | |
| Transformer VAE | 0.836 | |
| next-step prediction (different objective) | decoder-only | 0.636 |
| tail infilling (different objective) | GLM | 0.782 |
Put together, the picture is consistent: order flow carries modest, mostly-linear structure, enough for a tiny linear model to capture, not enough to justify a transformer.
9. Execution time on CPU (no CUDA)
Accuracy wasn’t the whole story here since everything runs CPU-only, so we also timed each model under identical conditions: the same 8-thread CPU, no GPU involved anywhere, the same seed-0 data (13,630 train / 3,407 validation windows), and the same 25-epoch recipe each model used for its own numbers above.
| Model | Params | Train (25 epochs) | Inference (256-window batch) | Inference (per window, batched) |
|---|---|---|---|---|
| Linear / PCA | 7,408 | 4.8s | 0.27 ms | ~1 μs |
| GRU | 46,603 | 111.7s | 24.6 ms | ~96 μs |
| Transformer AE | 45,863 | 225.0s | 19.3 ms | ~75 μs |
| Transformer VAE | 45,863 | 223.5s | 22.1 ms | ~86 μs |
| Decoder-only | 45,807 | 259.0s | 25.2 ms | ~99 μs |
| GLM | 45,839 | 261.1s | 22.9 ms | ~89 μs |
(Single-run wall-clock timings, meant to give a rough sense of relative cost, not a rigorously repeated benchmark.)
A few things stand out:
- Training time tracks architecture, not parameter count. The GRU has the most parameters of any model here (46,603) but trains roughly twice as fast as any transformer. Recurrence over a 32-step window is cheap on CPU; attention’s O(S²) cost dominates even at this small scale.
- The four transformer-based models cluster together, 223 to 261 seconds to train, regardless of what they’re being trained to do. The VAE’s extra KL term and the GLM’s masking logic add negligible cost next to the attention stack itself.
- The linear model is in a different league. About 45x faster to train and 70-90x faster at inference than any transformer here, at roughly a sixth of the parameters. Combined with Section 8’s result, that it also reconstructs best, the “simpler model wins” story in this piece isn’t just about accuracy. The model that wins is also the cheapest one to train and run, by a wide margin.
- Inference latency isn’t really the issue either way. Every model answers in well under a millisecond per window, transformers included. It’s the accuracy gap, not raw speed, that actually rules out the heavier models here.
10. What we actually take away
None of this is really about markets, it’s about method. A few things we’d carry into the next project:
- A latent has to be load-bearing, and learn to stay that way. Fixing the wiring wasn’t enough; the training dynamics collapsed it anyway. Free-bits, or a real downstream consumer, is what keeps a latent alive.
- Tell a sequence model about order. The models that worked were the order-aware ones. The original VAE’s missing positional encoding wasn’t a minor detail; it was disqualifying.
- Fit a baseline before reaching for sophistication. A linear model and a mean baseline cost minutes and reframed everything. We would have saved ourselves a lot of time by fitting them first.
- Match complexity to the data’s actual structure. Sophistication only pays off when the data has structure to reward it. Here it didn’t, and every step up the complexity ladder cost accuracy (and, per Section 9, time).
- Statistically real isn’t the same as real. Across the wider project, promising numbers repeatedly evaporated under a proper control, a shuffled label, an out-of-sample split, a realistic constraint. We caught more than one near-false-positive mid-celebration. Trying to disprove your own result is what separates research from wishful thinking.
And one last point: a clean negative result, carefully arrived at, is still worth writing up. “We built the sophisticated thing, tested it fairly against the simple thing, and the simple thing won” isn’t a story to bury.
Every model here is tiny (~46k parameters, the linear baseline far smaller), trains on CPU in minutes, and is backed by unit tests for each fix. No GPU, no CUDA anywhere, reproducibility was part of the point.