Start here for the limits of behavioral logs, then read feedback to objectives: missingness versus negatives, multi-signal supervision, and how to test whether a signal helps.
For the training perspective, start with Why post-training keeps coming back to data. One question with clear, conflicting, or missing evidence connects data selection to SFT, RL, environment checks, and evaluation.
Logs record behavior, not complete intent
Training logs record what users did with the choices, interface, and exposure available at the time. They provide clues, not a complete record of intent or preferences.
This matters because we routinely mistake “what happened in the logs” for “what the user actually wanted.”
Start with one click
Suppose a recommender has never shown you jazz, only pop.
You click a few pop songs. The log will say:
user saw pop songs → user clicked pop songs
But it cannot prove:
user only likes pop
Jazz never entered your choice set. The system chose what to show, so the log only contains responses to those options. Here, policy bias and exposure bias refer to this recommender setting: the display policy affects which feedback can be observed. Sequence generation has an exposure bias too, with the same name and a different meaning: teacher forcing trains on the correct prefix, while generation continues from the model's own outputs; see teacher forcing in Seq2Seq.
What one log entry actually mixes together
Tracing the example makes the missing feedback visible:
flowchart TB
pool["Candidates: pop and jazz"] --> shown["System shows only pop"]
shown --> clicked["User clicks a pop song"]
clicked --> log["Log: one click on pop"]
pool -. "Jazz was not shown" .-> missing["No feedback on jazz<br/>Not evidence of dislike"]
So data is not a clean label. It is the joint product of several mechanisms.
The five most common problems
1. Sparsity
Most people do not explicitly like or explain every result. Missing feedback may mean dislike, but it may also mean the user never saw the result, had no time, lost the connection, or already got the answer from the title.
2. Delay
Some outcomes show no value in the moment. A search may only help the user finish a task days later, and a piece of advice may need long-term use before anyone knows whether it fit.
3. Policy bias
The system only receives feedback on what it chose to show. The more it relies on old logs, the more likely it is to repeat the blind spots of the old policy.
4. Missing longitudinal structure
A single click is easy to record; connecting events across time is harder. If an assistant is meant to help over the long term, it needs to track changing needs rather than just store disconnected events.
5. Ambiguous labels
A long dwell may mean the content was valuable, or that it was hard to understand. A like may be for the opinion, the author, the image, or simply a polite response.
What better data design looks like
A good data system does not just store more fields. It spells out the semantics of each observation:
- What happened: what did the user, the model, the tools, and the environment each do?
- Why it was visible: which retrieval, ranking, or product policy produced the exposure?
- When the feedback occurred: was it an immediate reaction or a delayed outcome?
- How strong the evidence is: an explicit correction usually carries more information than one swipe.
- Which objective this record can support: SFT, preference learning, RL, or evaluation only?
- How to reproduce it: which model, prompt, candidate set, feature, and data versions were used?
More data is not more information
A million highly correlated clicks may be worth less than a hundred interactions that carry explicit context and a reason for the correction.
What matters is whether the data adds new information about the goal, the user, or the environment, not whether it adds rows.
How it connects to other topics
- Representation and memory decides how this data becomes long-term state.
- Search affects what users see and therefore which feedback the system can collect.
- Post-Training decides how this feedback changes model behavior.
- Evaluation checks whether the model learned useful information or mainly reproduced biases in the data.
- Human-in-the-Loop can provide fewer but clearer correction signals.
The questions I now ask first
When I get a dataset, I do not start by asking how many rows it has. I ask:
- Which policy produced this data?
- Which people, content, or failures never had a chance to be recorded?
- How far is the observed signal from the real objective?
- Does it have temporal order and context?
- If a model optimizes this signal, which loophole is it most likely to exploit?