Case study · AI & Data Science Program · Project 2

Lead-Conversion Prediction

A model that finds 85% of the leads who will pay, and the signals behind it: time on the website, a website first touch, and referrals.

  • 85%of converting leads found (test recall)
  • 4,612leads, about 30% converted
  • 67.7%referral conversion (63 of 93)
  • 7 minon the site: the strongest signal
0.50.70.90.50.751.0Recall: buyers found →Precision →Tree, defaultTree, prunedForest, defaultForest, tuned (chosen)
Referrals
67.7% (63 of 93)
Every other channel
about 28–32%

Conversion rate among leads who saw each channel, on a 0–100% scale.

4,612 leads, about 30% converted. Test results for the converted class; the tuned forest finds 85% of converters, and about two in three leads it flags convert.

A missed buyer is lost revenue; an extra call costs minutes.

So I tuned for recall: the tuned forest finds 85% of buyers, and about two in three leads it flags convert.
On this page

Problem

An online-learning startup gets more leads than sales can call. It needs a ranked call list.

Data: 4,612 leads, 14 attributes, about 30% converted. Program certificate (external site).

Result / Impact

What drives conversion

  • Time on the website is the strongest signal; the pruned tree splits at about 414–420 seconds, roughly 7 minutes.
  • A website first touch and high profile completion convert best.
  • Referrals convert at 67.7% (63 of 93), more than twice any other channel (about 28–32%), but are only 93 of 4,612 leads.

What I’d tell the business

  • Sales: call website-first leads with 7+ minutes on the site first. The model catches 85% of eventual buyers; about two in three leads it flags convert.
  • Marketing: grow referrals and make profile completion easier. Product: find out why app leads convert less.

What I did

The split, the tuning and the four-model comparison
  1. Split 70/30, stratified (3,228 training, 1,384 test leads).
  2. Tuned for recall on converters with cross-validated grid search and balanced class weights: a missed buyer is lost revenue; an extra call costs minutes.

Test results, converted class (balanced class weights):

Model Accuracy Recall Precision F1
Tree, default 0.79 0.63 0.66 0.64
Tree, pruned (depth 3) 0.78 0.89 0.58 0.70
Forest, default 0.86 0.72 0.79 0.76
Forest, tuned (100 trees, depth 5) 0.83 0.85 0.67 0.75

The tuned forest’s train/test gap is 2 points (0.85 vs. 0.83); the overfit default’s was 1.00 vs. 0.86. All four were compared on the same test set, so its scores are somewhat optimistic.

Tools: Python, Pandas, NumPy, scikit-learn, Matplotlib, Seaborn, Google Colab.

Decisions

The tuned forest, and a simple tree to explain it

The tuned forest over the highest-recall tree. Context: sales time is scarce, but missed buyers cost more. Options: most accurate (default forest: finds 72% of buyers); highest recall (pruned tree: 89%, but only 58% of flagged leads convert); balanced (tuned forest). Decision: the tuned forest: 4 points less recall than the tree for 9 more of precision, and 13 points more buyers than the most accurate model. Consequences: one flagged lead in three won’t convert; set the threshold with sales, using real call costs.

Keep the simple tree for explanation. Context: sales won’t act on a black box. Options: feature importance only; also a depth-3 tree. Decision: the tree explains (“website first, 7+ minutes”); the forest predicts. Consequences: the explanation is approximate, and I say so.

What I’d do next

  • Tune the threshold against real costs; try calibrated gradient boosting; trial the list for a month.
  • Check time on site for leakage (sessions after a sales call).

Design, not experience

How I’d run this in production on Databricks (design, not production experience)

CRM and web events streamed into Delta; one lead feature table for training and scoring; MLflow runs, promoted only if recall at a fixed precision improves; hourly scoring into the CRM; alerts when recall drops.

Next case study

  • AI & Data Science Program · Project 3

    0.8808test RMSE, tuned SVD

    Product recommender

    Four recommenders compared on 65,290 ratings, and a metric blind spot that changed the design.

Need a Claude agent or data workflow built? Work with me