Feature store: when the served value stops matching training

A feature store has one job: the value a model sees at training time has to be the same value it sees in production. When that breaks, the model gives a correct answer to a question that has changed.

The trouble is that nothing visibly breaks. No exception, no alert, no failed job. Offline metrics stay excellent, because offline everything is consistent.

Two paths for one feature

The same feature is computed twice: once in batch for training, once in real time for inference. The two paths run on different infrastructure and are often written by different people at different times.

The three forms of skew

1. Timestamp drift

Offline, we ask for the value of the feature at the time of the event. Online, we ask for its value now. If the online store refreshes every five minutes, a request arriving at 14:03 reads a value computed at 14:00. The training set never contained that delay: the point-in-time join was exact.

So the model learned on fresh features and serves on features that can be up to five minutes old. The gap is invisible offline, and it grows when the pipeline falls behind, which means when the system is under load.

What you see: accuracy drops at peak hours and comes back at night, with no model change at all.

2. Backfill mismatch

A backfill recomputes history with today's logic. That is often the point. It is also how a feature picks up information that did not exist back then.

Take a "customer lifetime value" column recomputed with the current definition. It includes purchases that had not happened yet at the event date of the row. The model learns from the future, posts great offline scores, and has nothing to stand on in production.

It is data leakage that looks like a quality improvement. It gets through code review because the change looks like a bug fix.

What you see: a nice offline improvement after a pipeline change, and nothing moving online.

3. Schema coercion

The warehouse and the online store disagree on types, and one of them converts without telling anyone.

A DECIMAL(10,2) in the warehouse becomes a float64 offline, a string in the online JSON payload, then a float32 at serving time. A null becomes NaN on one side and 0.0 on the other. A category encoded by sorted index offline gets a different index online, because the sort ran over a different set of values.

That last case is worth saying plainly: a category encoder fitted at training time and refitted at serving time becomes a different function. A fitted encoder is a model artifact. It belongs next to the weights, and it should never be recomputed in a preprocessing step that runs twice.

What you see: a small, constant accuracy gap that no retraining manages to close.

The maths, intuitively: the age of a served value

Timestamp drift comes down to a single quantity: the age of the value returned by the online store, meaning the time since it was computed.

If the store refreshes every T minutes, that age resets to zero at each refresh and then climbs to T. That gives a sawtooth. A request arrives at any moment, so on average in the middle of a tooth:

mean age = d + T/2

where d is the pipeline delay, between the event and the write to the store. Meanwhile the true value keeps moving. The gap between what the model receives and what it should have received is roughly the rate of change of the feature times the age: gap ≈ slope × age.

With a refresh every 5 minutes and a 1-minute delay, each request reads a value that is 3.5 minutes old on average. Offline, the point-in-time join gives an age of zero, so training never saw this gap.

Two practical consequences. For a slow-moving feature the gap is negligible. For a fast-moving one, like a click counter or a price, it matters. And it grows when the pipeline falls behind, since d goes up at peak hours: that is the symptom from the first section.

How to detect them

Looking at one path on its own does not help. Each one is consistent with itself, and the disagreement only shows up when you compare them.

Log what was actually served. You need the exact feature vector that went into the model, at request time, with the request ID. Every check below depends on it, and it is the step teams skip most often, usually because of storage cost. A sample is enough: 1% of traffic already finds systematic skew.

Replay offline from that log. Recompute the same features through the offline path, for the same entities at the same timestamps, and compare. A difference on a feature that should be deterministic is skew, by definition.

class=class="tok-str">"tok-com"># Skew check: recompute through the offline path, compare with what was served.
class=class="tok-str">"tok-com">#
class=class="tok-str">"tok-com"># We compare column by column: an accuracy drop says something is wrong,
class=class="tok-str">"tok-com"># this script says which feature is the culprit.

import pandas as pd

served = pd.read_parquet("logs/served_features/dt=class="tok-num">2026-class="tok-num">03-class="tok-num">23")  # request_id, entity_id, event_ts, features
offline = feature_store.get_historical_features(
    entity_df=served[["entity_id", "event_ts"]],
    features=FEATURE_LIST,
).to_df()

merged = served.merge(offline, on=["entity_id", "event_ts"], suffixes=("_online", "_offline"))

for col in FEATURE_LIST:
    a, b = merged[f"{col}_online"], merged[f"{col}_offline"]
    mismatch = ~((a == b) | (a.isna() & b.isna()))
    if mismatch.any():
        rate = mismatch.mean()
        print(f"{col}: {rate:.class="tok-num">2%} mismatched")
        print(merged.loc[mismatch, [f"{col}_online", f"{col}_offline"]].head())

This script has to run on a schedule. Skew comes with changes, and changes keep coming.

Compare distributions when equality makes no sense. Some features are not deterministic, for example anything derived from a timestamp. For those, compare distributions and alert when the summary statistics shift.

Make freshness visible. I emit the age of every online value as a metric. Without that number you cannot reason about timestamp drift, and it is almost always higher than the schedule suggests.

Fixing it

One definition for both paths. The fix that lasts is a single declarative definition of the feature, from which both paths are generated. Two hand-written implementations will drift apart, the only question is when. That is why feature store frameworks exist, and it helps to remember it when the abstraction feels heavy.

Ship fitted transformers with the model. Encoders, scalers, imputers: fitted at training time, serialized, loaded at serving time. A transformation whose parameters are learned from data is part of the model.

Test point-in-time correctness. A small synthetic dataset with a known right answer, run through the offline path on every pipeline change. Point-in-time joins are tricky enough that a test beats a careful review.

Treat the served-features log as production data. Retention, schema, monitoring. It is the only reliable record of what the model actually saw.

Why it is hard

The two sides pull in different directions. The offline path is written by people who care about correctness over the whole history. The online path, by people watching p99 latency. They meet on a schema, and a schema cannot say "as of the event" or "now".

Each path is correct on its own. Nothing raises an exception. The only signal is a model that does worse than expected, which is easy to mistake for a model that is just average. That is the first diagnosis most teams reach.

The defense is simple and unglamorous: log what you serve, replay it, compare, alert on the gap. The rest of the architecture helps you avoid the problem. To know whether it is there, you have to measure it.