Modeling Item Selection and Relevance for Accurate Recommendations more

Modeling Item Selection and Relevance for Accurate Recommendations: A Bayesian Approach ACM International Conference on Recommender Systems (ACM RecSys) 2011 Nicola Barbieri*+ Gianni Costa+ Giuseppe Manco+ Riccardo Ortale+ *University of Calabria - Department of Electronics, Informatics and Systems for High Performance Computing and Networks (ICAR) Italian National Research Council +Institute Wednesday, October 26, 2011 Outline • • Prediction vs Recommendation accuracy • • • • • Evaluating Recommendation accuracy Probabilistic approaches to Recommendations Latent Factor Modeling Performance of Probabilistic approaches on Top-N Recommendations • • • Bayesian User Community Model Model Specification Parameter estimation Experimental Evaluation Conclusion Wednesday, October 26, 2011 Prediction vs Recommendation Accuracy • • The recommendation problem has been traditionally interpreted as a missing value prediction problem (matrix completion) Standard approach: minimize statistical error metrics (MSE, RMSE) • • The common belief is that small improvements in prediction accuracy would reflect an increase of the accuracy of the recommendation lists However, recent works have shown that there is no monotonic relation between error metrics and accuracy metrics: lower RMSE does not imply higher recommendation accuracy (Cremonesi et al., RecSys 2010) Wednesday, October 26, 2011 Evaluating Recommendation accuracy • • • Standard definitions of Precision & Recall We assume that a recommendation meets user satisfaction if he/ she can find in the recommendation list at least an item which meets his/her interests Following the methodology proposed by Cremonesi et al. Wednesday, October 26, 2011 Probabilistic Approaches to Recommendation • • • Each triple ⟨u, i, r⟩ is considered as the output of a random observation drawn for the joint distribution of the random variables U, I and R Bag of words assumption: preference observations are assumed to be generated independently Two main modeling perspectives: • • Forced Prediction: focus on the estimate of P(r|u, i) • • Rank by expected preference value Free Prediction: the item selection process is included in the model, which is typically based on the estimate of P (r, i|u) If we assume that the selection is independent from the rating, P(r, i|u) can be factorized as P(r|i, u)P(i|u) • • Item selection Item Selection and Relevance Wednesday, October 26, 2011 Probabilistic Approaches to Recommendation: Advantages • • • • • • • Representation via graphical model They do not focus on a particular error metric: parameters are determined by maximizing the likelihood of the data They model a distribution over rating values Possibility to plug prior knowledge into the generative process No regularization terms to avoid overfitting They provide an unified framework for combining collaborative and content features Combining item selection and rating prediction, they achieve the best recommendation accuracy (New!!! (Barbieri and Manco, EMCL PKDD 2011)) Wednesday, October 26, 2011 Modeling Free Prediction User Community Model (Barbieri et Al. SDM 2011) Wednesday, October 26, 2011 Modeling Free Prediction Bayesian User Community Model Wednesday, October 26, 2011 A Bayesian User Community Model • • Each user is modeled as a random mixture of topics Each topic is characterized both by a distribution modeling itempopularity and by a distribution over preference values Wednesday, October 26, 2011 A Bayesian User Community Model • • Each user is modeled as a random mixture of topics Each topic is characterized both by a distribution modeling itempopularity and by a distribution over preference values Wednesday, October 26, 2011 A Bayesian User Community Model • • Each user is modeled as a random mixture of topics Each topic is characterized both by a distribution modeling itempopularity and by a distribution over preference values Wednesday, October 26, 2011 A Bayesian User Community Model • • Each user is modeled as a random mixture of topics Each topic is characterized by a distribution modeling itempopularity and by a distribution over preference values Wednesday, October 26, 2011 Model Specification • • • The Bayesian formulation is better suited to the sparsity of the rating matrix and less susceptible to overfitting due to the prior modeling Given the hyperparameters, the complete data likelihood can be computed as: By rearranging the components and grouping conjugate distributions: Wednesday, October 26, 2011 Parameter Estimation • • • • The posterior distribution of hidden variables can be specified as Unfortunately, exact inference is intractable We devised a Gibbs Sampling procedure The probability of observing the topic k for the observation <u,i,r> is directly proportional to the number of times that: • • • an item evaluated by u has been assigned to topic k the item i has been assigned to topic k the rating r has been assigned to the item i when the topic is k Wednesday, October 26, 2011 Inference Procedure Hyperparameters can be updated employing Minkaʼs iterative approximation Convergence measured on Held-out LogLikelihood Wednesday, October 26, 2011 Experimental Evaluation • We study the performance of the BUCM on Movielens1M data and a sample of the Netflix dataset, according to two perspectives: • • Prediction accuracy Recommendation accuracy: employing Item Selection and Relevance as ranking function Wednesday, October 26, 2011 Predictive accuracy • We evaluate the RMSE of Bayesian UCM over the MovieLens data set and compare its predictive accuracy against a selection of probabilistic competitors • URP-Gibbs vs Bayesian UCM: • • URP-Gibbs: the detection of topic is aimed to increase the likelihood of observing a similar rating behavior within the topic itself Bayesian UCM: the generic topic gathers those users who tend to assign similar ratings to items that are frequently experienced within the same community Wednesday, October 26, 2011 Recommendation Accuracy (Movielens) Wednesday, October 26, 2011 Recommendation Accuracy (Netflix) Wednesday, October 26, 2011 Recommendation Accuracy: Considerations • • BUCM achieves the best performance against all the competitors, and in general the two datasets confirm the same trend As expected, the gain in accuracy with respect to LDA is more significant when US-precision and US-recall are taken into account Wednesday, October 26, 2011 BUCM vs URP • • • Similar generative process and inference based on Gibbs Sampling URP, though exhibiting a higher predictive accuracy than Bayesian UCM, poorly performs in terms of recommendation accuracy The simultaneous modeling of item selection and rating patters, only yields a reasonable overhead in typical scenario (#latent factors<30) Wednesday, October 26, 2011 Conclusions • • • • BUCM provides a Bayesian reformulation of the former UCM approach The underlying idea is to assume a generative process, which simultaneously takes into account both item selection and rating emission Item selection for free prediction tends to have a negative impact on the resulting predictive accuracy, but Bayesian UCM outperforms state-of-art approaches to in terms of recommendation accuracy Wednesday, October 26, 2011 Conclusions • • • • • BUCM provides a Bayesian reformulation of the former UCM approach The underlying idea is to assume a generative process, which simultaneously takes into account both item selection and rating patterns Item selection for free prediction tends to have a negative impact on the resulting predictive accuracy, but Bayesian UCM outperforms state-of-art approaches to recommendation in terms of recommendation accuracy Is it possible to balance prediction and recommendation accuracy? Wednesday, October 26, 2011 Thanks Wednesday, October 26, 2011