Case study · AI & Data Science Program · Project 3

Product Recommendation System

Four recommenders compared on the same data, and a finding the metrics hid: the answer is a layered design, not a single winner.

  • 0.8808test RMSE, tuned SVD (best)
  • 65,290ratings, 1,540 users, 5,689 products
  • 0.866F1@10, tuned SVD
  • 4.29KNN fallback average, above the 3.5 threshold
12345relevant ≥ 3.5fallback 4.29

RMSE on the test split (lower is better)

SVD, tuned
0.8808
Best KNN
0.9526

Scale 0 to 1.0. F1@10: 0.866 (SVD) vs. 0.870 (user-user KNN). 65,290 ratings · 1,540 users · 5,689 products.

KNN’s fallback predicts the global average (4.29), above the 3.5 relevance threshold: fine in the metrics, random to the shopper.

No generic guesses shown as personal.

A KNN model with too few neighbors returns the global average (4.29), which clears the 3.5 threshold, so I drop those predictions and backfill from SVD.
On this page

Problem

An electronics store wants to recommend products each shopper will rate highly, including new shoppers with no history.

Data: public Amazon electronics ratings (about 7.8 million, per the dataset description), filtered to users with 50+ ratings and products with 5+: 65,290 ratings, 1,540 users, 5,689 products. Ratings skew high (mean 4.29, median 5). Program certificate (external site).

Result / Impact

A layered recommender, not one model:

  • Tuned SVD as the main personalized model: best rating accuracy (RMSE 0.8808 vs. 0.9526 for the best KNN), F1@10 of 0.866, and always a real personal estimate.
  • Item-item KNN for “customers who liked this also liked…”, only where it has real neighbors; SVD fills the rest.
  • Popularity for new shoppers, with a minimum-ratings rule.

The finding that changed the design. When a KNN model can’t find enough neighbors, it doesn’t fail: it returns the global average (4.29) and flags the prediction “impossible”. The tuned user-user model did this for both my example pairs; the tuned item-item model’s top five for one shopper were all this fallback. Since 4.29 clears the 3.5 relevance threshold, these guesses count as “relevant”: fine in the metrics, random to the shopper.

What I did

Four models, tuning and the full comparison table

Built popularity, user-user and item-item KNN, and SVD, each tuned by 3-fold grid search on RMSE, and compared them on an 80/20 split (top-10 metrics; relevant = true rating 3.5+). The grid searches used the full filtered data, so the folds overlap the test split and tuned scores are somewhat optimistic.

Model RMSE P@10 R@10 F1@10
User-user KNN, baseline 1.0012 0.855 0.858 0.856
User-user KNN, tuned (k = 50, cosine) 0.9526 0.847 0.894 0.870
Item-item KNN, baseline 0.9950 0.838 0.845 0.841
Item-item KNN, tuned (k = 30, MSD) 0.9576 0.839 0.880 0.859
SVD, baseline 0.8882 0.853 0.880 0.866
SVD, tuned 0.8808 0.854 0.878 0.866

Tools: Python, Pandas, Surprise (KNNBasic, SVD, GridSearchCV), Matplotlib, Seaborn, Google Colab.

Decisions

SVD first, and dropping impossible KNN predictions

Lead with SVD, though user-user KNN had the top F1@10. Context: F1@10 ranked user-user first (0.870 vs. 0.866); RMSE ranked SVD first. Options: top-F1 model; SVD; models by role. Decision: models by role, SVD primary. Consequences: 0.004 lower F1 for real personal estimates everywhere.

Drop “impossible” KNN predictions. Context: they look relevant in the evaluation. Options: keep them; drop and backfill from SVD. Decision: drop and backfill. Consequences: fewer item-item suggestions on sparse products; no generic guesses shown as personal.

What I’d do next

  • Add a “predict the average” baseline every model must beat; evaluate on a time-based split; measure coverage and novelty; A/B test the design.

Design, not experience

How I’d run this in production on Databricks (design, not production experience)

Ratings into Delta; scheduled retraining (Spark’s ALS at full scale) logged in MLflow, promoted only if better; precomputed top-N lists; alerts when the fallback share rises.

Next case study

  • HackerRank Orchestrate · Bronze medal · Claude agents

    Bronzerank 363 of 4,090

    AI agent engineering

    518 data-deletion requests audited in 41 minutes. The analysis came before the prompt.

Need a Claude agent or data workflow built? Work with me