Yelp ML Platform

A model that scores well in a notebook is not a service, and the distance between the two is where most of the actual work hides. The Yelp Open Dataset is 5.3 GB of raw JSON across 6.99 million reviews, which is past the point where a single-machine dataframe stays comfortable and past the point where the training code and the serving code can honestly be the same code. The aim was to take the raw dumps all the way to a REST API answering two unrelated questions, which business to recommend and how a review actually feels, and find out what breaks in between.

Spark is the right tool for the first half of that and the wrong tool for the second. It converts 5.3 GB and factorises 4.39 million interactions without complaining, but the same session called from inside a web request spends roughly 290 ms on a single prediction, almost all of it planning a query and crossing into the JVM to score one row. The data pushes back as well: Yelp star ratings pile up at the top end, so a three-class sentiment model can post a healthy accuracy while barely handling the middle class, and recall on a recommender is a small number by nature (any honest figure looks like failure until a baseline sits next to it).

The shape follows from that split. Everything heavy happens once, offline, and the request path is allowed to know nothing about Spark.

• One pass over the raw JSON. The ETL writes snappy Parquet in a single Spark pass at about 462,000 rows per second on one eight-core node, then 5-core filters to 4.39 million interactions over 287,000 users and 148,000 items, so every job downstream reads a columnar table instead of re-parsing JSON.
• Train in Spark, serve without it. The sentiment model is exported to plain numpy artifacts (the TF-IDF vocabulary, the IDF vector, the logistic-regression coefficients) and the transform is reimplemented on top of them, which takes p99 to 0.11 ms and about 34,000 predictions per second.
• The rejected option was keeping Spark in the request path. It is one code path rather than two and it cannot drift, but 290 ms per single-row prediction is not a serving story, and a dedicated model server meant another runtime to install and version for what is, in the end, a sparse dot product.
• Bias terms before bigger models. ALS on its own predicts ratings poorly when the ratings are this skewed, so global, user and item bias terms went in first and took RMSE to 1.17, which is a cheaper win than widening the latent space.

Two implementations of the same maths is a genuine liability: the numpy path can drift from the trained model and nothing would visibly fail, it would just answer differently. The fix is to make parity a test rather than a claim, checking that the exported artifacts reproduce Spark's predictions on held-out reviews, which currently holds at 100 percent. The rest of the tax is ordinary but not free: MLflow to keep runs and model versions straight, Docker Compose so the API and its dependencies come up the same way twice, a Pytest suite in CI, and benchmark numbers written to provenance-stamped files that are committed rather than quoted from a terminal.

The serving side is the clean result: 0.11 ms at p99 against roughly 290 ms for the in-process Spark path, at full prediction parity. The models are more modest and are worth stating plainly. Recall@10 of 5.5 percent means nothing on its own, but it is 6.2 times the most-popular baseline it was measured against, and 86.3 percent sentiment accuracy sits beside a macro-F1 of 0.70 against a 0.67 baseline, which is the class imbalance showing through. Both are reported against those baselines below rather than in isolation.