Tiny BERT in Production: Millisecond Text Classification on a CPU

Four independent classifiers converged on the same recipe: a 2-to-4 layer BERT, quantized with OpenVINO, running in single-digit milliseconds on a machine with no GPU. This post covers how, and why the way you split your data mattered more than the model.

The constraint

Most NLP writing assumes you have a GPU, or that latency is someone else’s problem. Ours was not. The target was a real-time pipeline that had to classify short documents as they arrived, on commodity hardware:

• A 4-vCPU cloud instance (Intel Sapphire Rapids class), no GPU
• A steady stream of documents, each needing a verdict in real time
• Classification runs on the document body only, with no clean title or metadata to lean on

That budget rules out a lot. We know because the first version of this pipeline was a large language model. It ran on a hosted Llama 3.3 70B Instruct model, queried over a serverless REST endpoint with a multi-part, retrieval-augmented prompt. It worked, but it took 1 to 5 seconds per document, and every call went to infrastructure we did not control and paid for by the token.

That latency is not a tuning problem. It is the nature of the approach. A generative model produces its answer one token at a time, and each token is a full forward pass through 70 billion parameters, wrapped in a network round-trip in the hosted case. Even in-house, and setting the LLM aside, a 110M-parameter bert-base runs at about 40 to 70ms per inference on this CPU. When the whole processing budget is a few milliseconds, seconds per call is not something you optimize away.

Latency was not even the main problem. We drove the endpoint with guided JSON, a schema that pinned every field’s type and format (a pattern-constrained identifier, a bounded number, a boolean), so the model had to return a well-formed object instead of free prose. On paper that guarantees a parseable answer. In practice, a serverless hosted model is a shared, best-effort service. Latency moved unpredictably across that 1 to 5 second range, and the responses varied in quality, and occasionally in validity, from one identical call to the next. Guided decoding constrains the shape of the output. It does nothing for the reliability of the service delivering it. Between the variable latency and the variable output, this was not something we could put in a real-time hot path. We needed a different approach, and one we controlled.

The fix came from reframing the problem. The task was originally posed as extraction: read the document and emit the relevant entity. That is a generation problem, which is why an LLM was the natural first tool. But generation is the expensive path. The entity we needed was almost always already present in the text, recoverable by a cheap probability-based lookup over the candidates the document itself contained. That reduced the model’s job to a single question: is this document relevant, yes or no? That is short-document binary classification, one forward pass, no generated tokens.

Reframing generation as classification helped in two ways. The first was speed: a multi-second generative call became a single-digit-millisecond one, two to three orders of magnitude faster. The second was accuracy. The generative approaches, the LLM and an early sequence-to-sequence model that also tried to emit the entity, were both less accurate than the binary classifier that replaced them. A model that only has to answer one yes/no question makes fewer mistakes than one composing a string token by token, where any token can be wrong. Getting both speed and accuracy is uncommon. Reducing the scope of what the model has to produce is one of the few ways to do it.

There is a third benefit that is easy to undervalue: control. The classifier runs on our own CPU, so it is deterministic and self-contained. No shared endpoint, no external variance, no rate limits, no per-token cost. The same document gives the same answer in the same few milliseconds every time. After the variable hosted endpoint, that predictability was worth as much as the speed.

One more detail is worth mentioning. The LLM was weak at emitting the entity, the harder generative sub-task, but reliable at the binary judgment of whether a document was relevant. So we used it for that: the 70B model produced the first-pass labels for the training set, which we then reviewed and corrected by hand. The LLM trained a model roughly a thousand times smaller to reproduce the judgment it was good at, and then left the hot path. The rest of this post covers how.

Part 1: The small BERT family and the sweep

Google published a family of small BERT checkpoints alongside the well-known base and large models. They are named by their shape:

bert_uncased_L-{layers}_H-{hidden}_A-{attention_heads}

So L-2_H-128_A-2 is 2 transformer layers, 128 hidden dimensions, and 2 attention heads, a small fraction of bert-base (L-12_H-768_A-12). These models are built for latency-bound deployment and are underused.

We swept eight of them on the same dataset, measuring average inference latency and macro-F1:

Model	Latency	F1 (macro)	Note
L-2_H-128_A-2	~11ms	0.81	Fastest
L-4_H-128_A-2	~16ms	0.75	More layers, lower F1
L-2_H-256_A-4	~17ms	0.85	Chosen for one classifier
L-6_H-128_A-2	~21ms	0.84
L-4_H-256_A-4	~22ms	0.89	Best accuracy/speed mix
L-8_H-128_A-2	~23ms	0.86
L-2_H-512_A-8	~37ms	0.86	Width is expensive
L-2_H-768_A-12	~70ms	0.87	Diminishing returns

Two results stand out.

First, more depth did not help. L-4_H-128_A-2 has twice the layers of L-2_H-128_A-2 and a lower F1 (0.75 vs 0.81). For a narrow model on short text, stacking more thin layers made optimization harder without adding usable capacity. Widening the model (H-128 to H-256) at shallow depth gave the largest single gain in accuracy. For short-document classification, width beat depth across most of this curve.

Second, the accuracy ceiling arrives early. Going from the roughly 4M-parameter models to the 100M+ L-2_H-768_A-12 moved macro-F1 from about 0.85 to about 0.87, two points, while raising latency 4 to 6 times. On this task the signal is mostly lexical and local. A small model captures it, and the larger model spends its extra capacity on noise.

The practical takeaway is to sweep the shape rather than reach for the biggest model the latency budget allows. The best option for these classifiers was L-2_H-256_A-4 or L-4_H-256_A-4, a few million parameters, depending on how each one weighed recall against speed.

Part 2: The data split that inflates your metrics

Before any of those F1 numbers mean anything, the cross-validation has to be right. This is what separates a real result from a misleading one, and it is easy to miss.

Our documents are not independent. Many refer to the same underlying entity, and documents about the same entity share vocabulary and phrasing. If you split folds naively, with StratifiedKFold over the samples, the same entity lands in both train and validation. The model does not have to generalize. It only has to recognize the entity it has already seen. The validation F1 comes back high and misleading.

The fix is to split by group, not by row:

from sklearn.model_selection import StratifiedGroupKFold

# groups = the entity each document belongs to
# every entity lands entirely in train OR entirely in validation, never both
splitter = StratifiedGroupKFold(n_splits=n_folds)
for train_idx, val_idx in splitter.split(X, y, groups=entity_ids):
    ...

StratifiedGroupKFold keeps the class balance across folds (stratified) and keeps any single entity entirely within one fold (grouped). The difference from plain StratifiedKFold was large. The naive split inflated the metrics enough to make them useless for model selection. Every architecture in the sweep above was scored with grouped folds, which is why the ranking is reliable.

The main point: when your samples share a group key, such as a user, an author, a document source, or an entity, group-aware splitting is not optional. It is the difference between measuring generalization and measuring leakage.

Part 3: Quantization

A 4M-parameter model in FP32 is already small, but two things remained: the PyTorch runtime carries overhead, and FP32 leaves speed unused on a CPU with AVX2 integer instructions. This is where OpenVINO and quantization come in.

We exported each production model to OpenVINO IR in three precisions and benchmarked all three: FP32, INT4 (weight-only), and INT8 (static, with weights and activations quantized using a calibration set of a few hundred real documents).

For one of the classifiers (a L-2_H-256_A-4 model):

Precision	F1 (macro)	Core inference	End-to-end	On-disk
FP32	0.9976	4.78ms	6.89ms	39 MB
INT4 (w-only)	0.9964	4.42ms	6.47ms	11 MB
INT8 (static)	0.9976	5.92ms	7.96ms	11 MB

The main result: INT8 static quantization reduced the model from 39 MB to 11 MB with no measurable change in F1 (0.9976 to 0.9976). Static quantization is often considered the riskier option because it quantizes activations as well, so it needs a representative calibration set and can lose accuracy if that set is off-distribution. Here, calibrating on a few hundred real documents held accuracy steady while reducing the footprint by about 3.5 times.

INT4 weight-only was the fastest on core inference and the smallest, at a cost of about one ten-thousandth of F1 (0.9976 to 0.9964), small enough to ignore and a reasonable choice when latency is the binding constraint. A second, smaller classifier showed the same pattern: 19 MB in FP32 to 5.9 MB in INT8, with accuracy unchanged.

One detail caught us out: whether INT8 is faster than FP32 depends on the CPU. On an older machine without integer-matmul acceleration, INT8 static was slightly slower than FP32 (5.92ms vs 4.78ms core). The quantize and dequantize work around each operation cost more than the integer math saved, so the only benefit was footprint. On a CPU with AVX-512 VNNI (Intel’s INT8 matmul acceleration), the result reverses: the integer path is hardware-native, and INT8 static becomes the fastest option as well as the smallest. Quantization speed depends on both the model and the hardware. Choose INT8 static when the deployment CPU can accelerate it, and benchmark on the hardware you will actually deploy on, not your laptop.

One more factor matters: the runtime language. Everything above was measured through a Python inference path, which carries interpreter and glue overhead. So we built the hot path a second time in native C++, a full port of the inference pipeline (preprocessing, tokenization, the OpenVINO forward pass, and the deterministic post-steps), with its own test suite and a three-path parity check that confirms the C++ output matches Python to within 1e-4. On the target cloud CPU it measured about 2.5 to 2.6ms warm, roughly half the Python path. For a model this small, the framework overhead is no longer small next to the model itself. The Python wrapper can cost as much as the forward pass.

That port is not abandoned. It is paused, and the reason is the useful part. A native rewrite pays off once the thing you are rewriting has stopped changing. Ours has not yet. The pipeline is still growing, with more classifiers being added and the shared preprocessing still in flux, and every change has to be mirrored across two languages and kept at bit-parity. Maintaining that against a moving target means spending most of the time re-syncing rather than shipping. So the C++ build stays parity-tested and ready, and we will bring it back into production once the inference path settles. When the design stops changing, the native speedup is there to take. Rewriting a component in native code is something to do after its design settles, not before.

The result: production inference runs in the sub-8ms end-to-end range from Python today, with a parity-tested native build at about 2.5ms held in reserve for when the pipeline settles. This runs on a CPU with no accelerator, well inside a real-time budget.

Part 4: Production latency vs the benchmark

Benchmark numbers are measured on a warm model, in a loop, with nothing else competing for the cores. Production is less controlled, so we log the wall-clock inference time of every classification and keep it. Here is what about 2,000 real inferences over eight months looked like, measured through the Python inference path. (The paused native C++ build from Part 3 measures roughly twice as fast.)

Metric (steady state)	Value
Median (p50)	~5.3 ms
Mean	~5.6 ms
p95	~7.8 ms
Share of runs at or under 7 ms	~72%
Share of runs at or under 20 ms	~80%

The steady-state median, about 5.3ms, with p95 under 8ms, matches the lab benchmark closely (6.9ms end-to-end). The quantized model performs in production about as it did in the harness. But the full distribution shows two things a benchmark loop does not.

First, cold start accounts for most of the tail. The first inferences after a process starts are much slower. The first 30 runs averaged about 170ms, and the slowest single call was over 2 seconds. This comes from graph compilation, a cold instruction cache, and lazy initialization, all landing on the first few documents. More than 70% of the runs slower than 100ms occurred in the first 10% of a process’s lifetime, against about 7% afterward. The fix is a warm-up inference at startup: run a dummy document through the model before entering the hot path, and the tail on real traffic largely disappears.

Second, bursts contend for cores. Even fully warmed up, a small fraction of inferences spiked. On a 4-vCPU machine, when several documents arrive at once they compete for the same cores, and a model that runs in 5ms alone can take 10 to 20 times that under a burst. This is a capacity-planning issue rather than a model issue, but it is invisible if you only ever measure one inference at a time. If p50 is the only number you track, you do not know your real latency.

The point is not that the benchmark was wrong. A benchmark measures the model, while production measures the system, including warm-up, scheduling, and contention. Keep the full distribution, not just the mean.

Part 5: Optimize for the right error

A classifier’s threshold is a product decision expressed as a number. The default of 0.5 assumes false positives and false negatives cost the same. They usually do not.

For these pipelines a missed positive was more costly than a false alarm. A downstream step could filter a false positive cheaply, but a missed event was gone. So we tuned the threshold on the F-beta curve with beta favoring recall (F2), rather than the F1-maximizing point:

from sklearn.metrics import fbeta_score, precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_true, y_scores)
# pick the threshold that maximizes F2 (recall weighted 2x precision)
f2 = (1 + 4) * precision * recall / (4 * precision + recall + 1e-9)
best_threshold = thresholds[f2[:-1].argmax()]

This connects back to model selection. We did not trade recall for small inference-time gains. When a faster model saved a couple of milliseconds but lost a few points of recall on the positive class, we kept the slower model. The latency budget was a constraint to meet, not a metric to maximize. Once you are under budget, more speed adds nothing and recall still matters.

Part 6: Four classifiers, one recipe

We built four separate binary text classifiers, at different times, for different tasks, and they converged on the same architecture and deployment stack. Not from a shared template, but because the same constraints pushed each of them to the same place:

1. Small BERT (2 to 4 layers, 128 to 256 hidden), never the base model. The signal was local and lexical, and capacity beyond a few million parameters added noise rather than accuracy.
2. Grouped cross-validation, always. Each of these domains had entity-clustered documents, so each would have been misled by naive splitting.
3. OpenVINO INT8 static for production. About 3.5 times smaller, accuracy unchanged, single-digit-millisecond inference with no GPU.
4. Recall-weighted thresholds, because in each case a miss cost more than a false alarm.
5. bf16 for training (fp16 as fallback), never fp32 by default. Faster training with no accuracy cost, and a small enough GPU memory footprint to sweep architectures cheaply.

That convergence is the main finding. When four independent problems reach the same design, the design reflects the problem class, short-document, latency-bound, entity-clustered classification, rather than any single dataset.

Summary

If you are building a real-time text classifier on hardware without a GPU:

• Start small and sweep the shape. Try the Google small-BERT family (L-2_H-128 through L-4_H-256) before anything larger. Width often beats depth on short text. Let the accuracy/latency curve choose the model, not intuition.
• Split by group, not by row. If samples share an entity, author, or source, use StratifiedGroupKFold. Naive splitting gives you a high but meaningless F1.
• Quantize with OpenVINO. INT8 static (with a real calibration set) or INT4 weight-only usually costs close to zero F1 and shrinks the model 3 to 4 times. INT8 is fastest when the CPU has integer-matmul acceleration (AVX-512 VNNI). Benchmark on the hardware you will deploy on, because the answer changes with the CPU.
• Consider the runtime language, but rewrite late. For a model this small, Python glue can cost as much as the forward pass. A parity-tested native C++ build roughly halved our latency (about 2.5ms). We keep it ready but not yet in production, because a native rewrite only pays off once the pipeline has stopped changing. Until then you are maintaining bit-parity across two languages against a moving spec. Move to native code after the design settles.
• Tune the threshold for your actual error costs. The default of 0.5 is rarely right. Pick the F-beta that matches which mistake is more expensive.
• Train in half precision by default.
• Warm up at startup and log every inference. A warm-up call removes the cold-start tail, and logging the wall-clock latency of every request is the only way to see how far apart your p50 and p99 are.

None of this is unusual, which is the point. A few million parameters, a leak-free data split, and a good CPU runtime give you production-grade text classification in single-digit milliseconds, with no GPU and no external API.

Based on four production classifiers built on the Google small-BERT family with Transformers, Optuna for hyperparameter optimization, and OpenVINO for CPU inference.