Why your feature store is silently lying to you

Train/serve skew does not live in the model. It lives in the translation layer between the offline and online paths — and it fails silently, which is what makes it expensive.

A feature store exists to guarantee one property: the value a model sees at training time and the value it sees at inference time are the same value. When that guarantee breaks, the model is not wrong — it is answering a question nobody asked.

The breakage is almost always silent. No exception, no alert, no failed job. Offline metrics stay excellent, because offline is where the lie is consistent.

Two paths, one promise

The offline and online paths compute the same feature by different routes, on different infrastructure, usually written by different people at different times.

Offline and online feature paths diverging: the offline path does a point-in-time join over a warehouse, the online path reads the current value from a key-value storeevent streamsource of truthwarehousefull historykey-value storecurrent value onlypoint-in-time join"value as of event time"batch · minutes · SQLonline lookup"value now" · ms · Go/Pythontraining setlabels + featuresserved requestfeatures → model↕ every difference between these two paths is skew
Figure 2 — The two paths answer subtly different questions: "what was the value at event time" versus "what is the value now". Every difference in timing, aggregation window, null handling or type coercion between them becomes skew the model never sees offline.

The three skew patterns

1. Timestamp drift

The offline path asks "what was this feature's value at the moment of the event". The online path asks "what is this feature's value now". If the online store is updated on a five-minute schedule, a request arriving at 14:03 reads a value computed at 14:00 — a staleness the training set never contained, because the point-in-time join was exact.

The model learned on features with zero staleness and serves on features with up to five minutes of it. The gap is invisible offline and grows with pipeline lag, which means it grows precisely when the system is under load.

Symptom: accuracy degrades during traffic peaks and recovers at night, with no model change.

2. Backfill mismatch

A backfill recomputes history with today's logic. That is usually the intent — and it is also how a feature acquires information that did not exist at the time.

A "customer lifetime value" column backfilled with today's definition encodes outcomes that had not happened yet at the row's event time. The model learns from the future, scores brilliantly offline, and has nothing to work with in production.

This is leakage wearing the costume of a data quality improvement, and it survives review because the change looks like a bug fix.

Symptom: a sharp offline improvement after a pipeline change, with no corresponding online movement.

3. Schema coercion

The warehouse and the online store disagree about types, and one of them silently converts.

A DECIMAL(10,2) in the warehouse becomes a float64 offline and a string in a JSON payload online, parsed back to float32 at serving. A null becomes NaN in one path and 0.0 in the other. A categorical encoded by sorted-unique-index offline gets a different index online because the sort ran over a different set of values.

The last one is worth stating plainly: a category encoder fitted on training data and re-fitted at serving is a different function. Fitted encoders are model artefacts and belong next to the weights, not in a preprocessing step that runs twice.

Symptom: a small, constant accuracy gap that no amount of retraining closes.

How to find them

You cannot detect skew by looking at either path alone. Both are internally consistent; the disagreement only exists between them.

Log what was actually served. Not what you believe was served — the exact feature vector that entered the model, at request time, with the request id. This single practice makes every check below possible and is the one most often skipped, usually on storage-cost grounds. Sample it if you must: 1% of traffic is enough to find systematic skew.

Replay offline against those logs. Recompute the same features through the offline path for the same entities at the same timestamps, and diff. Nonzero diff on a feature that should be deterministic is skew, by definition.

class=class="tok-str">"tok-com"># Skew check: recompute through the offline path, compare to what was served.
class=class="tok-str">"tok-com">#
class=class="tok-str">"tok-com"># The comparison is per-feature, not on the model output: an aggregate
class=class="tok-str">"tok-com"># accuracy gap tells you something is wrong, this tells you which column.

import pandas as pd

served = pd.read_parquet("logs/served_features/dt=class="tok-num">2026-class="tok-num">03-class="tok-num">23")  # request_id, entity_id, event_ts, features
offline = feature_store.get_historical_features(
    entity_df=served[["entity_id", "event_ts"]],
    features=FEATURE_LIST,
).to_df()

merged = served.merge(offline, on=["entity_id", "event_ts"], suffixes=("_online", "_offline"))

for col in FEATURE_LIST:
    a, b = merged[f"{col}_online"], merged[f"{col}_offline"]
    mismatch = ~((a == b) | (a.isna() & b.isna()))
    if mismatch.any():
        rate = mismatch.mean()
        print(f"{col}: {rate:.class="tok-num">2%} mismatched")
        print(merged.loc[mismatch, [f"{col}_online", f"{col}_offline"]].head())

Run it on a schedule, not once. Skew is introduced by changes, and changes are continuous.

Assert on the distribution, not just on equality. Some features are legitimately non-deterministic — anything derived from a timestamp, for instance. For those, compare distributions rather than values, and alert on a shift in the summary statistics rather than on any difference.

Make staleness a feature you can see. Emit the age of each online feature value as a metric. You cannot reason about timestamp drift without knowing how stale things actually get, and the number is almost always worse than the schedule suggests.

Fixing it

One definition, two executions. The durable fix is a single declarative feature definition from which both paths are generated. Two hand-written implementations will diverge; the only question is when. This is what feature store frameworks are for, and it is worth remembering when the abstraction feels heavy.

Ship fitted transformers with the model. Encoders, scalers, imputers: fitted at training, serialised, loaded at serving. If a transformation has parameters learned from data, it is part of the model.

Make point-in-time correctness testable. A synthetic dataset where the correct answer is known, run through the offline path on every pipeline change. Point-in-time joins are subtle enough that reasoning about them is not sufficient.

Treat the served-feature log as production data. Retention, schema, monitoring. It is the only ground truth about what the model actually saw.

Why this is so hard

Every incentive points the wrong way. The offline path is written by people optimising for correctness over full history; the online path by people optimising for p99 latency. They meet at a schema, and a schema does not carry semantics — it cannot express "as of event time" versus "as of now".

The failure is silent because both paths are individually correct. Nothing throws. The only signal is a model that works less well than it should, which is indistinguishable from a model that is simply not very good — and that is the diagnosis teams reach for first.

The practical defence is narrow and unglamorous: log what you served, replay it, diff it, and alert on the diff. Everything else is architecture that helps you avoid the problem, which is worth having and is not a substitute for measuring it.