Problem
An online-learning startup gets more leads than sales can call. It needs a ranked call list.
Data: 4,612 leads, 14 attributes, about 30% converted. Program certificate (external site).
Result / Impact
What drives conversion
- Time on the website is the strongest signal; the pruned tree splits at about 414–420 seconds, roughly 7 minutes.
- A website first touch and high profile completion convert best.
- Referrals convert at 67.7% (63 of 93), more than twice any other channel (about 28–32%), but are only 93 of 4,612 leads.
What I’d tell the business
- Sales: call website-first leads with 7+ minutes on the site first. The model catches 85% of eventual buyers; about two in three leads it flags convert.
- Marketing: grow referrals and make profile completion easier. Product: find out why app leads convert less.
What I did
- Split 70/30, stratified (3,228 training, 1,384 test leads).
- Tuned for recall on converters with cross-validated grid search and balanced class weights: a missed buyer is lost revenue; an extra call costs minutes.
Test results, converted class (balanced class weights):
| Model | Accuracy | Recall | Precision | F1 |
|---|---|---|---|---|
| Tree, default | 0.79 | 0.63 | 0.66 | 0.64 |
| Tree, pruned (depth 3) | 0.78 | 0.89 | 0.58 | 0.70 |
| Forest, default | 0.86 | 0.72 | 0.79 | 0.76 |
| Forest, tuned (100 trees, depth 5) | 0.83 | 0.85 | 0.67 | 0.75 |
The tuned forest’s train/test gap is 2 points (0.85 vs. 0.83); the overfit default’s was 1.00 vs. 0.86. All four were compared on the same test set, so its scores are somewhat optimistic.
Tools: Python, Pandas, NumPy, scikit-learn, Matplotlib, Seaborn, Google Colab.
Decisions
The tuned forest over the highest-recall tree. Context: sales time is scarce, but missed buyers cost more. Options: most accurate (default forest: finds 72% of buyers); highest recall (pruned tree: 89%, but only 58% of flagged leads convert); balanced (tuned forest). Decision: the tuned forest: 4 points less recall than the tree for 9 more of precision, and 13 points more buyers than the most accurate model. Consequences: one flagged lead in three won’t convert; set the threshold with sales, using real call costs.
Keep the simple tree for explanation. Context: sales won’t act on a black box. Options: feature importance only; also a depth-3 tree. Decision: the tree explains (“website first, 7+ minutes”); the forest predicts. Consequences: the explanation is approximate, and I say so.
What I’d do next
- Tune the threshold against real costs; try calibrated gradient boosting; trial the list for a month.
- Check time on site for leakage (sessions after a sales call).
Design, not experience
How I’d run this in production on Databricks (design, not production experience)
CRM and web events streamed into Delta; one lead feature table for training and scoring; MLflow runs, promoted only if recall at a fixed precision improves; hourly scoring into the CRM; alerts when recall drops.