AI Product Modernization Playbook · 09 / 12
A scoring model becomes useful when it helps a product make a better decision: which case deserves review, which customer may benefit from an offer, or which application needs more evidence. Its value depends on the target, the decision policy and what happens after the score is used.
Imagine a customer-success team receiving a ranked list of accounts likely to leave. The list looks convincing, but the team has a limited retention budget. Some high-risk customers will leave regardless of the offer; others would stay without it. Predicting the outcome accurately does not by itself reveal which intervention will change that outcome.
This chapter of the AI Product Playbook explains how to build risk and propensity scoring into an existing product. It connects trustworthy labels, probability calibration, explicit policies, explanations and controlled evaluation. Credit, retention and investigation workflows illustrate the design choices, while the level of validation and review should reflect the consequences of each use.
On this page
1. Start with a decision contract
Define the decision before choosing an algorithm. Name its owner, eligible population, available actions, operating capacity and consequences of errors. Prioritizing a sales queue, referring a claim for investigation and changing access to credit have different costs and requirements even if they all use a probability between zero and one.
Specify the score's meaning
State the event, time horizon and conditions. “Probability of cancellation within 90 days among active monthly subscribers under the current service policy” is more useful than “churn score.” A default score likewise needs an explicit delinquency definition, observation window and population. A lead score should clarify whether it predicts a response, a qualified opportunity or a completed purchase.
Define what the product should do when data is missing, the case is outside the validated population or evidence is too weak. These states should not silently become a low score or an unfavorable decision. Record the action limits and circumstances that require referral, independent review or a new assessment.
Choose a measurable decision outcome and guardrails: qualified conversions per unit of sales effort, incremental retained margin, useful findings per investigator hour, or approvals evaluated alongside loss and customer outcomes. Establish the existing policy as a baseline so a better model score is not confused with a better operating result.
2. Audit the target and the outcome window
The label determines what the model learns. Cancellation, non-renewal and inactivity are different outcomes. A late payment is not identical to eventual loss. An investigation finding reflects both underlying behavior and what investigators chose to examine. Document how labels are created, corrected and linked to the original decision.
Wait for outcomes to become observable
A recent account that has not yet completed its outcome window is not automatically a negative example. Mark incomplete follow-up explicitly. Use mature cohorts for fixed-horizon evaluation, or suitable time-to-event methods when censoring and timing matter. Check the assumptions behind those methods, including whether loss to follow-up is related to risk.
Preserve intermediate events separately from the final target. Early arrears, reduced usage or failed renewal attempts may be useful monitoring signals, but they should not silently replace the event used to validate the production model. A changed target definition creates a new model-use contract.
A convenient proxy can encode unequal treatment
Obermeyer and colleagues' 2019 study examined a health-management algorithm that predicted healthcare spending as a proxy for need. At the same score, Black patients were sicker than White patients because spending did not reflect needs equally. The authors estimated that remedying the disparity would increase the share of Black patients selected for additional help from 17.7% to 46.5%. This was an analysis of allocation, not a trial showing that a deployed replacement achieved those outcomes. The product lesson is to examine what the label represents, even when predictive accuracy looks strong. See the original study.
3. Reconstruct the information available at decision time
Build features from the data actually available when the score would have been issued. Keep event time, availability time, source version and lineage. A ticket written after cancellation, a later collection action or an investigator's final assessment cannot be an input to the earlier prediction.
Split development and evaluation along time, and separate entities when testing generalization to new customers or organizations. Fit preprocessing, feature selection and calibration within the appropriate training stages. Repeated observations from the same account can otherwise leak across a seemingly clean split.
Account for the policy that produced the dataset
Repayment outcomes are typically observed for approved loans, not for every declined applicant. Sales outcomes may be better observed for leads that received attention. A model trained on these records describes a selected population. It cannot establish performance on excluded cases merely by assigning them scores.
Reject inference and other missing-outcome techniques depend on assumptions; inferred outcomes are not observed ground truth. Evaluate sensitivity to those assumptions and restrict the approved use when evidence is insufficient. Any additional exploration must be appropriate for the domain and authorized; it should not casually expose people to harmful decisions just to improve a dataset.
Log exposure to offers, calls and reviews as well as outcomes. Without that history, the next training set may mix customer behavior with the effects of the previous policy. The point-in-time discipline described in the forecasting chapter applies here too.
4. Separate scoring, policy and final action
The model estimates an outcome. A separately versioned policy applies eligibility rules, thresholds, costs, capacity and action limits. The workbench records what a reviewer or authorized process actually decided. This separation allows each part to be tested and changed without losing the history of the others.
Make a past decision reconstructable
Store the score identifier, target and horizon, feature snapshot reference, model and calibrator versions, probability or explicitly labeled ranking score, and validity period. Link the record to the policy version, rule hits, proposed action, final action, reviewer, explanation and subsequent corrections or challenges.
Retain the artifacts and inputs needed to reproduce a decision under the applicable retention policy. A version label alone is insufficient if the associated code, feature transformations or external data disappear. Where exact replay is constrained by lawful deletion or inaccessible historical services, define the retained evidence and limitation explicitly rather than promising indefinite reproducibility.
Keep evaluation attributes and sensitive monitoring data under appropriate access controls. The production feature interface should expose only inputs authorized for the specific use, with tenant boundaries and purpose restrictions enforced before scoring and explanation.
5. Test ranking and probability quality separately
Begin with an interpretable baseline such as logistic regression or a well-designed scorecard, then benchmark more flexible candidates where the data supports them. Monotonic constraints can encode justified domain relationships, but they do not automatically guarantee fairness, causality or stable explanations. Validate the constraints as assumptions.
Interpret calibration as a cohort property
A calibrated prediction near 0.20 means that approximately 20% of comparable cases receiving such predictions experience the defined event over the specified horizon. It is not a guarantee about one person. Check observed versus predicted rates by score band, time and material segment, with sample sizes and uncertainty.
Fit any sigmoid or isotonic calibrator on predictions from cases not used to train the underlying model, while respecting time and entity separation. Keep a final evaluation set outside model, calibration and threshold tuning. Representative prevalence matters: resampling or class weighting can distort probability interpretation unless addressed and validated on the deployment population.
Use reliability diagrams alongside proper scoring rules such as Brier loss or log loss. Those losses assess more than calibration alone, so an improvement does not prove that every band is better calibrated. See the scikit-learn calibration guide. Prioritize the regions that drive consequential decisions without hiding failures elsewhere.
A ranking-only score can support a validated prioritization task, provided the interface identifies it correctly. Probability-based economics need probability validation. Avoid presenting an arbitrary score of 80 out of 100 as an 80% likelihood.
6. Choose policies using costs, constraints and capacity
A model's default classification threshold is rarely a complete operating policy. Evaluate candidate cutoffs against the relevant costs of false positives and false negatives, the available review capacity and required safeguards. The right action can also depend on exposure, customer context and uncertainty, not only the predicted event rate.
Define what happens in each decision band
For a lead queue, a band may trigger routine outreach, specialist review or no current action. For a consequential assessment, bands may route cases for additional evidence or an authorized decision. Give each route a reason, owner and service expectation. A queue that grows faster than reviewers can handle changes the effective policy, even if model latency remains low.
Evaluate the complete population affected by each policy. Holding loss rate constant while changing the approval population can still change total loss, margin and access. Report both rates and volumes, together with the uncertainty of estimates. Do not select the most flattering denominator.
Route cases with invalid data or unsupported use to an explicit exception path. Proximity to a cutoff can be a useful review signal, but it is not itself a statistical confidence interval. Test how small plausible input changes affect the decision, and distinguish sensitivity from uncertainty about the outcome.
7. Use intervention effects for retention and offers
Propensity predicts an outcome under the conditions represented in the data. Intervention targeting asks a different question: how would that outcome change if a particular action were taken? A purchase probability is also distinct from the treatment-assignment propensity used in causal estimation; use precise names in the data model and product documentation.
Compare incremental value, not just churn risk
Consider two hypothetical cohorts. Cohort A has an 80% churn probability without an offer and 78% with it. Cohort B has 40% without the offer and 30% with it. If each incrementally retained customer contributes $200 and treatment costs $10 per contacted customer, the simplified expected net values are minus $6 for A and plus $10 for B.
The arithmetic assumes causal response estimates, a fixed treatment cost for every contacted customer and a contribution measure excluding that cost. Real policies also need eligibility, contact limits, uncertainty, long-term effects and budget constraints. Two ordinary propensity models trained on selectively offered customers do not automatically identify the causal difference.
Randomized experiments can estimate response differences when their design and operating conditions support the comparison. Observational approaches require additional assumptions, including adequate overlap and control of relevant confounding. Evaluate the learned targeting policy on independent data. Eva Ascarza's Retention Futility uses two field experiments to show that the customers with the highest churn risk need not be the best retention targets; it does not imply that every offer works or that uplift estimates are certain.
8. Explain the decision users actually received
The workbench should show the target and horizon, current evidence, relevant reasons, data quality, policy rule and next step. Distinguish reasons affecting the model estimate from rules that determined the final action. A hard eligibility rule should not be explained as though it were a feature attribution.
Test reason quality and stability
Map supported reasons to clear wording and verify their fidelity to the actual model and policy. Feature attribution can be unstable when inputs are correlated; it is not a causal explanation of what will happen if a user changes something. Counterfactual suggestions need feasible changes, realistic dependencies and the full decision policy. They should not promise a future outcome the system cannot guarantee.
Give reviewers access to source evidence and a structured way to correct data, request more information or override a recommendation. Record the reason and assess outcomes on comparable cases. Overrides are selected interventions, so their raw average performance should not be treated as proof that humans outperform the model.
New York's Department of Financial Services reported in March 2021 that its Apple Card investigation did not produce evidence of deliberate or disparate-impact discrimination, while identifying customer-service and transparency deficiencies. That finding should not be rewritten as proof of unlawful bias or as a universal guarantee of fairness. It demonstrates that explanations and support processes need their own validation. See the regulator's report.
9. Evaluate fairness across the full workflow
Removing a protected attribute does not remove all potential bias. Other inputs can encode related information, the target can reflect unequal historical treatment, and the policy can distribute review effort or benefits unevenly. Examine data collection, labels, scoring, thresholds, overrides and outcomes together.
Choose measures for the actual harm
Review decision rates, relevant error rates, calibration and user outcomes across appropriate groups and intersections where the data can be lawfully used. Include sample sizes, missing group information and uncertainty. An aggregate result can hide a small subgroup with poor outcomes, while a small sample can make a reported gap unstable.
Fairness criteria can conflict, and no single ratio establishes legal compliance or ethical acceptability. Define the objective and tradeoffs with the responsible business, domain and legal teams; record what remains unequal and why. Treat thresholds such as an approval-rate ratio as contextual review tools, not universal safe harbors.
Distinguish data authorized for auditing from features authorized for decisions. Sensitive data may be necessary for a lawful evaluation, with safeguards and restricted access; its use is a domain-specific decision. Proxy tests are diagnostic tools, not an automatic rule to remove every correlated feature. Evaluate legitimate purpose, alternatives and the actual effects of the complete system.
10. Make review and challenge meaningful
Map the applicable privacy, automated-decision and sector requirements before implementation. Specify when a decision can be automated, when a person must review it, what information must be provided and how people can correct errors or challenge the result. These obligations depend on the use and jurisdiction; a generic “human in the loop” label does not resolve them.
In SCHUFA, C-634/21, the Court of Justice held that generating a credit score can fall within automated individual decision-making under the GDPR when the recipient gives it a determining role in granting credit. That finding is about the score's actual role, not a blanket claim that every score is prohibited.
In Dun & Bradstreet Austria, C-203/22, the Court explained that meaningful information must describe the procedure and principles actually applied so the person can understand which data was used and how. Merely supplying an algorithm is insufficient. The product implication is to retain the evidence needed for understandable, case-specific explanations and review.
Reviewers need time, relevant evidence and authority to change the outcome. Inspect review quality and escalation patterns rather than using a low override rate as proof of success. A language model may help draft explanations from verified reasons and approved wording, with checks for unsupported claims and omitted options. It should not invent the score, causal reasons or authority to make a consequential decision.
11. Validate the model and the decision policy
Use an untouched later-period evaluation and challenge the intended use with independent reviewers proportionate to its impact. Compare candidate models and policies against the current process on the same eligible populations, horizons and outcome maturity. A model performing well in shadow mode has not yet demonstrated the effects of changing real decisions.
Swipe horizontally to read all columns.
| Layer | Evidence to collect | Key limitation to check |
|---|---|---|
| Ranking | ROC-AUC, precision-recall performance, lift or precision at usable capacity | Aggregate ranking does not establish calibration, fairness or decision value. |
| Probabilities | Observed versus predicted outcomes by band; proper scoring rules | Inspect maturity, prevalence, segment sample sizes and drift. |
| Policy | Action rates, costs, loss or margin, referral workload | Report rates and totals; account for selection and operational capacity. |
| Interventions | Incremental outcomes and net value against a credible comparison | Separate response prediction from causal effect; include treatment costs and uncertainty. |
| Fairness and explanation | Group outcomes, errors, reason fidelity and review quality | No single metric or generic explanation establishes acceptable treatment. |
| Operations | Freshness, latency, replay, fallback, overrides and challenges | Test the entire path, including external data and exception handling. |
Stress-test relevant scenarios and slices: new acquisition channels, incomplete histories, economic changes, missing external data and correlated applications. Record the bounds of the validated population. Choose acceptance criteria using local baselines and consequences rather than copying universal numerical gates from a template.
Keep validation for reason wording, policy changes and calibration changes alongside model evaluation. A more readable explanation can still be wrong, and an unchanged model can create different outcomes after a cutoff or limit changes.
12. Roll out with mature outcomes and explicit fallbacks
Start with a parallel run against the current policy, followed by controlled use where appropriate. Separate the effect of changing the model from the effect of changing eligibility, limits or offers. Randomized comparisons can help when permissible and well designed; other designs need their assumptions and limitations documented. A challenger label alone does not make a comparison unbiased.
Monitor more than input drift
Track score distributions, data quality, action rates, referral load, reason patterns, overrides and complaints immediately. Compare predicted and observed outcomes once cohorts mature, by time and relevant segment. Early indicators can reveal trouble before the final label arrives, but they do not prove long-horizon performance.
Distribution metrics such as population stability index are diagnostic signals, not universal pass/fail rules. Investigate whether a change reflects growth, a new channel, missing data, gaming or a deteriorating relationship between features and outcomes. Recalibration cannot repair every form of model misspecification or selection bias.
When scoring or required data is unavailable, use an explicitly approved fallback such as deferred assessment, manual review or a validated existing policy. Display the reason and preserve the decision record. Avoid interpreting technical failure as evidence that the person or account is high risk. Test backlogs and recovery as well as the happy path.
Give one accountable owner responsibility for decision outcomes across data science, policy and operations. Maintain versioned release criteria, rollback procedures and periodic reassessment of the approved use. The monitoring patterns in the anomaly detection chapter can help detect problems in the scoring service itself.
13. Deliver a complete decision capability
A strong first release supports one decision and one defined population. Its delivery package includes the target contract, label and selection audit, baseline comparison, calibrated model where probabilities are needed, versioned policy, faithful explanations, review path and outcome monitoring.
Expand only when evidence shows that the full workflow improves the intended outcome within its constraints. For retention, that means measuring incremental response and value. For consequential assessments, it also means validating how the system treats people and how errors can be challenged and corrected.
Start an implementation discussion with the decision users make today, the outcomes you can observe, the actions available and the cost of being wrong. Those details determine whether a score is useful and what must surround it in the product.
Next in the series: computer vision—how to turn images and video into reliable product workflows, from capture quality to review and deployment.
Sources and further reading
This article develops the risk scoring and propensity chapter of the AI Product Playbook. The architecture and retention calculation are implementation guidance and illustrative assumptions, not measured Infinity Technologies client results. Legal examples describe specific published decisions; implementation requirements must be assessed for the actual use and jurisdiction.
- scikit-learn, Probability calibration, for reliability diagrams, calibration fitting and interpretation of scoring rules.
- Obermeyer and colleagues, Dissecting racial bias in an algorithm used to manage the health of populations, Science, 2019, for the effect of the target proxy.
- Eva Ascarza, Retention Futility: Targeting High-Risk Customers Might Be Ineffective, Journal of Marketing Research, 2018, for evidence on retention targeting.
- New York State Department of Financial Services, Report on Apple Card Investigation, March 2021.
- Court of Justice of the European Union, SCHUFA judgment announcement, December 7, 2023, and Dun & Bradstreet Austria judgment announcement, February 27, 2025.
