Evaluating Models for Real Decisions

A model can look excellent in a vendor demo and still send your reps to the wrong accounts.

The problem is usually not the algorithm. It is the scorecard. Many model evaluations answer a statistician's question: how well does this rank everything? Operators need a different answer: if we act on this list at the volume we can actually handle, do we make more money than we do today?

Accuracy is the wrong first question

Say 5% of your customers churn in a quarter. A model that predicts "nobody churns" is 95% accurate. It is also useless.

The next number most teams see is AUC, short for area under the ROC curve. In plain words, it is the chance that the model ranks a randomly chosen churner above a randomly chosen customer who stayed. It is a fair summary of ranking across the whole book. But you do not act on the whole book. You act on the top of it.

When the outcome is rare, like churners or closable leads, research in PLOS ONE found that ROC plots can be deceptive, and that precision-recall plots are more informative because they measure how many of the flagged cases are real.^[1]

Two terms are worth learning because they map straight to operating questions:

• Precision: of the accounts we flagged, how many actually churned? That is the share of CS calls that were worth making.
• Recall: of the accounts that churned, how many did we flag? That is the share of the problem we caught.

Evaluate at the cutoff you will actually use

Your CS team can make 40 save calls a week. Your SDRs can work 200 inbound leads a day. That capacity is the operating point, and it is where the model should be judged.

The useful question becomes: how good is the top 40? Compare it to 40 accounts picked the way you pick them today. The ratio is lift: how many times more real churners (or closed deals) you find per call by following the model.

Where you draw the line matters too. The documentation for scikit-learn, a popular open-source machine learning library, splits classification into two parts: predicting a probability, and deciding what to do about it. It calls the default 50% cutoff "most certainly not ideal for most use cases," and in its insurance example the tuned cutoff lands around 2%.^[2]

Cutoffs also go stale. A 2015 NeurIPS paper from Google engineers notes that manually set thresholds can become invalid when a model is retrained on new data.^[3] Re-check the cutoff every time the model changes.

Put dollars on the mistakes

A model makes two kinds of mistakes, and they rarely cost the same.

A false alarm on churn costs a CSM hour, and maybe a discount given to someone who was never leaving. A miss costs the account. On the lead side, a false alarm is a wasted call and a miss is a lost deal.

Charles Elkan's paper on cost-sensitive learning makes the key point: when mistakes carry different costs, it can be optimal to act as if something is true even when it is not the most likely outcome. His example is declining a large credit card transaction that is probably legitimate. His recommendation is to have the model estimate probabilities, then make the decision explicitly using the costs.^[4]

Here is an illustrative version. Picture a software company where a save call costs about $150 of CSM time, a lost account is worth $12,000 a year, and a save call works one time in five. A call is worth making whenever churn risk is above roughly 6%, not 50%. The math is one line: $12,000 times 20% times the churn probability has to beat $150.

Google's published rules for machine learning engineering agree: what matters is what you do with the prediction, and utility beats raw predictive power.^[5]

Judge the model by the calls it changes and what those calls are worth, not by a score on a slide.

Calibration: can you trust the percentage?

The dollar math above only works if the probabilities mean what they say. That property is called calibration. The scikit-learn documentation gives the plain version: among accounts a well-calibrated model scores near 80%, about 80% should actually turn out positive.^[6]

Plenty of popular models rank well and still state the odds wrong. A study presented at ICML 2005 found that boosted trees and support vector machines push predicted probabilities away from 0 and 1, while naive Bayes pushes them toward the extremes. It also found that after a correction step, boosted trees, random forests, and support vector machines gave the best probabilities.^[7] The fix exists. You only get it if someone checks.

The check fits on one page. Group accounts by predicted risk (0 to 10%, 10 to 20%, and so on) and compare each group's prediction to what actually happened. If the 30% bucket churns at 10%, your dollar math overstates the payoff three times over.

Backtest on time, not a random shuffle

Many quick evaluations shuffle history and hold out a random slice. That lets the model learn from next spring's customers and then get graded on last fall's.

Google's rules are direct about it: if you train a model on data up to January 5, test it on data from January 6 onward.^[5] Expect it to be somewhat worse on newer data, not radically worse. For business models, train on the first 18 months, score the next quarter, and see if the top of the list held up. The related trap, fields filled in after the outcome, is covered in Feature Engineering for Tabular Business Data. And no evaluation rescues bad data.

Keep scoring after launch

Evaluation is not a launch gate. It is a monthly habit, because the customers a model learned from stop looking like the ones in front of you.

Three checks cover most of it. Precision in the top of the list each month. The calibration chart each quarter. And the NeurIPS paper's simple test: predicted totals should roughly match actual totals, and a change in that gap is often the first sign the world moved.^[3]

A one-page model scorecard

1. Base rate: what share of accounts actually churn, convert, or renew.
2. Precision and recall at your real capacity, not at a default cutoff.
3. Lift over the way you pick accounts today.
4. Expected dollar value at the chosen cutoff, with both kinds of mistakes priced.
5. Calibration chart showing predicted versus actual rates.
6. Time-based backtest against a simple baseline.
7. Monitoring plan with an owner and a review date.

If a vendor or internal team cannot fill this in, you do not yet know whether the model helps.

If you want lead scoring, churn, or forecasting models evaluated this way on your own data, talk to Gamify Data. We fit models to the history you already have and measure them against the decisions your team makes every week.

Sources

1. Takaya Saito and Marc Rehmsmeier, "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets," PLOS ONE, March 4, 2015. https://doi.org/10.1371/journal.pone.0118432
2. scikit-learn developers, "Tuning the decision threshold for class prediction," scikit-learn 1.9.1 documentation, accessed September 24, 2026. https://scikit-learn.org/stable/modules/classification_threshold.html
3. D. Sculley et al., "Hidden Technical Debt in Machine Learning Systems," Advances in Neural Information Processing Systems 28 (NeurIPS 2015), December 2015. https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
4. Charles Elkan, "The Foundations of Cost-Sensitive Learning," Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence (IJCAI 2001), August 2001. https://cseweb.ucsd.edu/~elkan/rescale.pdf
5. Martin Zinkevich, "Rules of Machine Learning: Best Practices for ML Engineering" (Rules #25 and #33), Google for Developers, last updated August 25, 2025. https://developers.google.com/machine-learning/guides/rules-of-ml
6. scikit-learn developers, "Probability calibration," scikit-learn 1.9.1 documentation, accessed September 24, 2026. https://scikit-learn.org/stable/modules/calibration.html
7. Alexandru Niculescu-Mizil and Rich Caruana, "Predicting Good Probabilities with Supervised Learning," Proceedings of the 22nd International Conference on Machine Learning (ICML 2005), August 2005. https://doi.org/10.1145/1102351.1102430