Lecture 8 latent factor recommender systems
Netflix Prize
- The winning entry factorized the problem into global (baseline), regional (SVD), and local levels (collab filtering).
Global:
- Baseline estimation: Avg rating + avg rating of movie above/below avg + avg rating of user above/below avg.
Regional: (SVD/Matrix Factorization and latent factors)
Local Neighborhood (k-nearest CollabFiltering/NN)
The Global and Local Levels
So Global and Local together (with some problems) is:
We update this model to address the problems so that we replace $s_{ij}$ with some learned weights $w_{ij}$.
The error metric is RMSE so we minimize RMSE (or the square of that, SSE). So we just learn $w_{ij}$ such that SSE on training data is mnimized (by grad descent, adam, etc)
The Regional Level (SVD/Matrix Factorization and latent factors)
We want to embed our movies in a low dimensional space so people who like similar movies are embeded together in this space.
Do “SVD” on the Netflix utility matrix. (not really SVD - we call it Matrix Factorization- we ignore empty values for reconstruction error) (1)
How to estimate the missing rating of user x for item i? Multiply the corresponding factors.
Modified SVD / Matrix Factorization:
We cannot actually use SVD because of missing entries, instead we learn the values to fill the embedding matrices through a learning algorithm. We no longer require P,Q to be orthogonal/unit length.
I.E. predict $p$,$q$ using alternating least squares The model deals with overfitting as we choose higher and higher values of latent factors $k$. (2)
We use l2 regularization to deal with overfitting.
(3)
So in summary, the goal is:
We optimize it using alternating stochastic gradient descent (Alternating Least Squres ALS)
Extending Latent Factor Model to Include Biases
Quite simple, instead of only training to fit $r_{xi} = q_ip_x$, above, we also treat $b_x$, $b_i$ as learned values and fit with ALS:
So,
The winning team also incorporated some data insights, factoring in metadata by change in rating over time, rating edge.
The winning model was the above model parameterized on time + throwing in a lot of metadata features in and hoping that it will work.
In recommender systems, what is really important is the regularization. The winning model in Netflix prize had over a billion parameters.
(1) We do SVD because SVD gives us exactly what we want - minimum SSE and RMSE for reconstruction error. However, we must modify SVD so that it is only operating over existing entries.
(2) Overfitting occurs when model has too much free parameters (too much freedom) that it starts fitting noise.
(3) Recap: Regularization:
The ‘length’ to be penalized is the size of the parameteres that we use. If we are constrained to use rather small parameters for each value of p and q, the model better be very good and very optimized.
If lambda is large, the model doesnt care about the ‘error’ so much as making the size of the weights tiny. If I have tiny weights, then I cannot make big mistakes.
For users with sparse data, the error cannot be very high anyways (there are not that many error terms to sum over). Thus, the regularization focuses on penalizing picking p, q too large. For users with a large amount of rich data, the error term is relatively large relative to the regularization term. In this case the model will focus most on shrinking the error term.
Notes: 1) Note that the baseline predictor (mean + user bias + movie bias) is actually really, really good! NEVER underestimate the baseline predictor 2)
