Data Collection That Improves Models

Most companies do not have a data volume problem. They have a recording problem.

In Feature Engineering for Tabular Business Data, we said to start with the fields already in the CRM and add tracking later, if you need it at all. This post is about that "later." When you do change what gets collected, a few specific habits improve lead scores, churn lists, and forecasts far more than another integration or another fifty columns.

More rows are not the goal

Researchers at Google interviewed 53 AI practitioners working in high-stakes fields and found what they called data cascades: data problems that compound and cause damage downstream. Of those practitioners, 92% reported experiencing at least one. The researchers titled the paper with a line that will sound familiar to any operator: "Everyone wants to do the model work, not the data work."^[1]

The business translation is simple. The collection choices your sales ops and CS teams make this quarter set the ceiling on how good your models can be next year. Nobody can backfill history that was never written down.

Record the outcome, with a date, every time

A model learns from outcomes. In machine learning those are called labels: won, lost, renewed, cancelled, downgraded. Without them, as we covered in Bad Data = Bad Results, you are ranking vibes.

The common gaps are small and fixable. Deals left open forever instead of closed lost. Churn logged without a date. Cancel and downgrade lumped together. A lost reason typed as free text that nobody can count.

Google's published rules for machine learning engineering put it plainly: before building a model, track as much as possible in your current system, because if something might matter later, it is better to have the historical data now.^[2] Start logging clean outcomes today, even if the model is six months away.

Record the ones you did not work

This gap is easy to miss.

If reps only call the leads that look promising, you only learn outcomes for leads reps liked. The leads nobody called have no result at all. Train a model on that history and it learns your team's taste, not lead quality. Statisticians call this selection bias: the data you kept was chosen by the same judgment the model is supposed to improve.

Google's rules describe the software version. A common pattern is to log only what users actually saw, which makes the data useless for learning why something was not seen, because all the negative examples were dropped.^[2]

It gets worse once a model is live. A 2015 NeurIPS paper from Google engineers warns that a model may directly influence the selection of its own future training data. Their suggested mitigations include some amount of randomization, or keeping part of the data isolated from the model's influence.^[3]

For a sales or CS team, that means a small random holdout. A slice of leads or accounts gets the standard play regardless of score. Google's rules give a concrete version for filtering systems: mark 1% of traffic as held out and let it through unfiltered, so those examples become cleaner training data.^[2]

The holdout costs something. A few good leads get worked in ordinary order, and a few weak ones get a call they did not earn. That is the price of two things you cannot get any other way: unbiased training data, and an honest answer to whether the score beats business as usual. Budget for it.

If you only record what you chose to do, the model can only learn what you already believe.

Record what you knew at the time

CRMs overwrite. Industry gets corrected after the deal closes. Stage history disappears when someone edits the record. Company size is updated during onboarding. When you later train a model, the fields describe the account as it looks today, not as it looked when the decision was made.

Google's rules recommend saving the inputs used at the moment of scoring and logging them for training, and warn that data joined from another table may change between training and scoring.^[2] The operator version: turn on field history tracking, or take a dated snapshot of the key fields when a lead is created, when a deal changes stage, and at each renewal date. That is also the cleanest defense against the leakage problem from the feature engineering post, where a field filled after the win makes the backtest look brilliant and the live score useless.

Make labels trustworthy

Even carefully built datasets have wrong answers in them. A NeurIPS 2021 study of 10 widely used public benchmark datasets estimated an average of at least 3.3% label errors in their test sets, including at least 6% of the ImageNet validation set.^[4] Those datasets were built by researchers who care about labels. Your CRM was built by busy reps at quarter-end.

Three habits help:

• Replace free-text reasons with a short picklist, required at stage change.
• Put the field where the work already happens, not in a separate form nobody opens.
• Audit a small sample of closed records every month and fix the definitions, not just the rows.

Google's rules add one more: give each data field an owner and documentation, so someone knows where it came from and what it means.^[2]

A short collection checklist

1. Outcome and date on every lead, deal, and account.
2. Reason codes from a picklist, not free text.
3. The action taken, including who was not called or saved.
4. A random holdout that gets the standard play regardless of score.
5. Snapshots of key fields at the moment decisions are made.
6. An owner for every field a model depends on.

None of this needs a new data platform. Most of it is CRM settings, a few required fields, and a monthly review. It changes what the next model can learn.

If you want help deciding what to record so your lead and churn models get better every quarter, talk to Gamify Data. We start with the history you already have and tell you which small collection changes will pay off.

Sources

1. Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora Aroyo, "'Everyone wants to do the model work, not the data work': Data Cascades in High-Stakes AI," Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, May 2021. https://doi.org/10.1145/3411764.3445518
2. Martin Zinkevich, "Rules of Machine Learning: Best Practices for ML Engineering" (Rules #2, #6, #11, #29, #31, and #34), Google for Developers, last updated August 25, 2025. https://developers.google.com/machine-learning/guides/rules-of-ml
3. D. Sculley et al., "Hidden Technical Debt in Machine Learning Systems," Advances in Neural Information Processing Systems 28 (NeurIPS 2015), December 2015. https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
4. Curtis G. Northcutt, Anish Athalye, and Jonas Mueller, "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks," NeurIPS 2021 Track on Datasets and Benchmarks, 2021. https://arxiv.org/abs/2103.14749