Apache Spark:Movie Recommendation Engine
Dataset:MovieLens
Output:Jupyter Notebook (please display the output results)
I think I got the majority of the programming completed but my spark crash and Iam trying to fix it.
Requirements for exercise:
In this exercise, you will create a movie recommendation engine from the MovieLens data. You can download that data by clicking here:MovieLens.
1. Movie Recommendation Engine
a. Prepare Data
Load the data from the ratings.csv and movies.csv files and combine them on movieId. The resultant data set should contain all of the user ratings and include movie titles. The schema should look something like this.
b. Train Recommender
Using the data you prepared in the last step, create a movie recommendation model using collaborative filtering.Spark’s collaborative filtering documentationprovides a template for building and testing this model.
Before you train the recommendation model, split the data into a training dataset and a testing dataset using the randomSplit dataframe method. Use 80% of your data for training and 20% for testing.
After fitting your model using the training dataset, calculate the predictions on the test dataset and use the RegressionEvaluator to calculate the root-mean-square error of the model.
As a reminder,Spark’s collaborative filtering documentationwill be helpful in completing this task.
c. Generate top 10 movie recommendations
Using the recommendation model, generate the top ten recommendations for each user. Using the show method, print the recommendations for the user IDs, 127, 151, and 300. You should not truncate the results and so should call the show method like this recommendations_127.show(truncate=False).