Predicting Rare Events from a Growing Text History

Our first model scored 0.71 in cross-validation and 0.55 on a year it had never seen: a coin flip dressed up as a signal. The problem was predicting a rare future event for an entity, using only the information available up to a point in time, from a stream of documents that keeps growing. It is a timeseries problem wearing a text-classification costume, and most of the work was not in the model. It was in the data, and in not lying to ourselves about what the model had learned.

The system is deployed on a single c7i.xlarge CPU instance, with no GPU, and it works through thousands of new documents a day: ingesting them, extracting structured facts, updating each entity’s profile, and re-scoring. What follows is the machine-learning side of how it got there, the architectures we tried, the metrics they reached, and the findings that held up.

The problem is a timeseries, not a snapshot

Each entity accumulates a history. Documents arrive over months and years from several independent sources, each carrying a timestamp. The question we ask is forward-looking: given everything known about this entity up to a cutoff date, will a specific rare event happen to it in the future?

That framing has two consequences that shape everything else.

First, a single entity is not a single training example. The same entity looks different at different times, so we sample each one at several horizons before the event: twelve months out, six, three, and one. Each snapshot contains only the documents that existed at that cutoff. This turns one entity into several point-in-time examples and teaches the model to make the call earlier, on less information, rather than only when the outcome is nearly obvious.

Second, the label is rare. Positives are a small fraction of the population, and they are clustered: several examples can belong to the same entity. Both facts make the evaluation fragile, and both make leakage very easy to introduce without noticing.

Temporal leakage is the whole problem

If you take one thing from this post, take this. On a forward-looking prediction built from an entity’s history, the dominant risk is not underfitting or the wrong architecture. It is future information quietly reaching the model through the training data.

The failure modes are subtle:

• Post-cutoff events. A profile snapshot at t-6m must contain no document dated after t-6m. Easy to state, easy to violate when profiles are assembled from a store that also holds later events.
• Entity all-time fields. Any attribute computed over the entity’s full history (a total count, a “current” status, a latest value) encodes the future at every earlier cutoff. These fields have to be recomputed as of each cutoff, not read off the entity record.
• The label’s own vocabulary. The event we are predicting has characteristic language. If any of that language, or the names of the other parties involved, appears in a pre-cutoff document, the model learns to read the answer instead of predicting it.
• Cross-entity relationships. Links between entities that were discovered later must be gated by when they were first observable, not by when we happened to record them.

We built a dedicated auditor that runs over every generated dataset and checks all of these: no event past the cutoff, no outcome vocabulary, no other-party names, cross-entity links gated by first-seen date, and entity-level fields reconciled against the event-derived values. It runs before any training job.

The reason this matters shows up immediately in the numbers. A gradient-boosted model on structured features scored a cross-validation AUC of about 0.71, which looks like a real signal. Its out-of-sample AUC, trained on past years and tested on a later one, was 0.55, which is barely above a coin flip. The gap between those two numbers is the gap between measuring memorization and measuring prediction. Several of our early high scores turned out to be leakage that the auditor later caught. A cross-validation number you have not leakage-audited and tested on held-out time is not a result. It is a hypothesis.

The text space kept growing

The interesting engineering was in how the entity’s history got represented, and it went through five stages, each richer than the last.

1. Structured counts. The first representation was a feature vector: counts of event types in trailing windows, ratios, and a handful of derived numbers. This is what the gradient-boosted baseline consumed. It captured how much was happening, but nothing about what any of it meant.
2. Embeddings plus counts. We embedded the available text with a sentence-transformer, mean-pooled it to a single fixed vector, and concatenated it with the numeric features. This added a little semantic signal but blurred everything into one average.
3. Per-event documents. Each event became a self-contained document of around 512 tokens, with relevant prior context injected and gated by the event date. Terse one-line summaries became short paragraphs that explained why an event mattered.
4. Full timeline profiles. Each entity became one long chronological document, a profile followed by its complete dated history.
5. Semantic chunks. The timeline profile was split into chunks by signal category, with each chunk holding complete events of a single type and no event split across a boundary.

The move to chunks was forced by a hard limit. A useful profile is far longer than the 512 tokens a small BERT reads, and truncating it throws away exactly the older context that a point-in-time model needs. Rather than reach for a long-context model as the default, we split the profile into category-coherent chunks, encoded each chunk independently, and combined them. This keeps every chunk inside the model’s native window and, as a side effect, makes the input order-invariant across categories.

We did try the long-context route. A 4,096-token model and an 8,192-token model (the latter fine-tuned with QLoRA) both ran, but the longer-context model was too slow for the marginal accuracy it bought, so neither shipped. On this data, chunking a small model beat stretching a large one.

What actually moved the metric

We ranked architectures with grouped cross-validation, and we checked the whole pipeline the only way that means anything on a time-ordered problem: rolling-origin validation. Train on the past, test on a future window the model has never seen, then roll the cutoff forward and repeat, eight times. Here is the progression.

Representation	Model	AUC
Structured counts	Gradient-boosted trees	0.71 CV, 0.55 rolling-OOS
Document embeddings	Perceiver IO (cross-attention)	~0.89 CV
Semantic chunks	Multi-chunk BERT, max-pool	~0.99 CV

Two numbers in that table carry the whole story.

The first is the gap on the baseline: 0.71 under cross-validation, 0.55 on a future window. Counts of how much was happening looked like a signal when folds were shuffled, and turned into a coin flip the moment the model had to call a year it had not seen. That gap is the most important number in the project. It is the distance between memorizing the population and predicting the future, and it is invisible unless you hold out time, not rows.

The second is the jump from counts to text. Feeding the model what the events actually said, instead of how many of them there were, was the largest gain anywhere in the work. A Perceiver IO over per-document embeddings reached about 0.89. Splitting each history into semantic chunks and encoding them with a small BERT reached about 0.99 in cross-validation.

And 0.99 is precisely where you should stop trusting a number and start investigating it. A rare, forward-looking label plus a generous cross-validation split is the textbook setup for leakage and overfitting to manufacture a beautiful score. So we did the unglamorous thing: re-derived every feature strictly from timestamped events, re-ran, and watched cross-validation fall from 0.999 to 0.991. Most of the signal was real. Eight thousandths of it was the future leaking in. Hunting down that 0.008 took more effort than any architecture change in the project, and it mattered more than all of them.

With the numbers established, three findings from the sweep held up and saved time later.

Pooling: the simplest option won. The multi-chunk model encodes each chunk independently and then pools the per-chunk vectors into one entity vector. On a fixed backbone, max-pool ranked first, ahead of attention pooling, then top-k mean, then log-sum-exp, and the gap to the runner-up was not small. The task is whether any chunk contains a strong enough cue, so taking the strongest activation per dimension beats trying to learn a weighting over chunks.

Chunk count saturated early. Performance was flat above roughly 128 chunks per entity; the top configurations sat within a hair of each other, and accuracy only dropped when the budget fell well below that. Context past the saturation point bought nothing, which capped how much we ever had to encode per entity.

The peak was an overfitting artifact. On the largest dataset the model hit its best cross-validation AUC at the first epoch and then degraded as training continued. Reading a number off that best-epoch checkpoint would have been the same self-deception as ignoring leakage. Between the early peak, the baseline’s 0.71-to-0.55 collapse, and the leaks the auditor kept finding, the lesson was consistent: on a problem like this, a high validation number is an instruction to go find the mistake that produced it.

Once the pipeline passed rolling-origin validation, the production model was retrained on all available data. That is the standard final step: you prove the approach generalizes to unseen time, then give the shipped model every example you have. The cost is that the deployed artifact carries no fresh score of its own, because there is no held-out data left to score it on. The validation lives in the splits that came before it, not in the final weights.

Data beat architecture, every time

The clearest pattern across months of work: when a result stalled, the fix was almost always in the data, not the model. We spent real time sweeping backbones, pooling functions, and hidden sizes, and those choices moved the metric by a point or two. Rebuilding the input, adding the signals that were missing from the chunks, fixing a leak, or representing an event as a paragraph instead of a count, moved it by ten or twenty.

One experiment makes the point sharper than any sweep. Our positives were instances of a broad event, and some were “cleaner” examples than others. The intuitive move is to train only on the clean ones so the model learns a sharper target. We tried it, and AUC fell from about 0.86 to somewhere between 0.60 and 0.71. The reason is obvious in hindsight: the positives we removed did not disappear, they became negatives, and they looked almost identical to the positives we kept. We had asked the model to draw a boundary through the middle of a single cluster. The broader, noisier label was the better one. How you define the label is a data decision, and it moved the metric more than any backbone we tried.

The most expensive mistakes were long training runs on data that was quietly wrong: chunks missing a signal we thought was included, or a feature that leaked. The habit that came out of it is dull and effective. Before any multi-hour job, unit-test the composition of the dataset, run the leakage auditor, and do a dry run on a handful of examples. Confirm the data is what you think it is before you spend the compute. Split by entity, never by example, so no entity appears in both train and validation: sharing an entity across the split turns generalization into recognition.

Running it on a CPU

The deployed system is not a notebook. It runs continuously on one c7i.xlarge, a four-vCPU instance with no GPU, and processes thousands of documents a day. The daily loop ingests new documents from several sources, parses them into structured events, updates each entity’s knowledge graph and profile, and re-scores the entities whose histories changed.

Inference is the multi-chunk model exported to a portable runtime, taking a variable number of chunks per entity through a batched forward pass and pooling to a single score. A small model (two to four layers) is what makes this affordable: scoring an entity means encoding up to a hundred-plus chunks, so the per-chunk cost has to be tiny for the per-entity cost to stay reasonable on a CPU. The same constraint that favored small models for single-document classification favors them even more here, where each decision is many encodes, not one.

The recipe

If you are predicting a rare, forward-looking event from an entity’s accumulating history:

• Treat it as a timeseries. Sample each entity at several cutoffs before the event, and let every snapshot see only what existed at that cutoff. One entity becomes several point-in-time examples.
• Assume leakage until you have ruled it out. Post-cutoff events, entity all-time fields, the label’s own vocabulary, and late-discovered relationships are the usual culprits. Write an auditor and run it before every training job.
• Trust rolling-origin over shuffled cross-validation. A high CV AUC next to a low future-window AUC means you are measuring memorization. Our baseline was 0.71 shuffled and 0.55 on held-out time. Report the number from the future window.
• Put the effort in the representation. Text beat counts by a wide margin. Getting the events into meaningful language moved the metric far more than any change of backbone.
• The label is a data decision. A broader, noisier positive set beat a narrow, “clean” one, because the excluded examples became negatives that looked just like the positives. Define the target before you tune the model.
• Chunk instead of stretching context. Split a long profile into category-coherent chunks inside a small model’s window, encode each, and pool. Max-pool beat learned pooling here, and chunk count saturated early.
• Fix the data first. When a result stalls, look at the input before the model. Unit-test dataset composition and dry-run before long jobs.
• Split by entity, not by example, or your metrics measure recognition.

None of the models here are large. The whole system runs on a CPU that costs a small amount per month. The hard part was never the network. It was building a leak-free, point-in-time view of a growing history, and checking, repeatedly, that the model was predicting the future rather than reading it.