AIML - Recommendation Systems

 Two techniques used

  • Content Based Filterting
    • Based on content recommendation is given to similar users
    • If user1 watches Action & Adventure movies, similarly user2 also sees same. Then new movies watched by User1 will be suggested to User2 based on category of movie. 
    • eg: amazon online shopping
  • Collaborative based filtering
    • identifies behavior of user and categorizes users accordingly. New movies will be suggested based on the other users in group irrespective of the genre/category
    • eg: Netflix, Amazon prime

Movie genre(Action, Comedy, Adventure, Romantic)

Content based recommendation system design - Movie recommendation system

  • Import data from tmdb(movie details with overview)
  • from sklearn.feature_extraction.text import TfidfVectorizer
    • remove stop words, special chars, remove nan value with blanks
    • fit transform on movies OVERVIEW field to get sparse matrix
  • from sklearn.feature_extraction.text import sigmoid_kernel
    • Note: sigmoid transforms input between 0 to 1
    • Apply sigmoid on the sparse matrix(pass same matrix to both arguments to get all combinations of how overview1 related to other overviews)
  • Reverse mapping of indices and movie titles

Recommender systems

  • Collaborative filtering - Book recommendation system using K nearest neighbors
    • online shopping - suggestion to by accessories along with iPhone 
    • online movies -- suggestion based on user ratings
  • Content based filtering
    • Netflix - recommendation  based on movie type

 

  • Weighted average for each movie average rating
    •  W=(Rv + Cm)/(v+m)
      • W - weighted rating
      • R - Avg for movie (0 to 10) = Rating
      • v - #of votes
      • m-minimum votes required to be in top 250
      • C - the mean vote across whole report
    • Recommendation based on both weighted average & popularity
      • import minmax scaller
      • transform weighted average & popularity to scale them down to 0-5 values
      • Predict score based on tranfored values * %weightage to be given
      • order by score in descending
  • Correlation
    • Download movie & user ratings data
    • Get mean of the ratings by movie for movies with #userratings>100
    • Create pivot table with name moviemat with user id & title as column
    • Correlate moviemat with startwarsuserrating.
  • Nearest Neighbours
    • online shopping - suggestion to by accessories along with iPhone
    • online movies -- suggestion based on other similar user ratings
    • Download movie & user ratings data 
    • Get mean of the ratings by movie for movies with #userratingcount>100
    • Create pivot table with name moviemat with user id & title as column
    • Import NearestNeighbours package(This is unsupervised ML)
    • Create variable by initiating NearestNeighbours object passing parameters metric 'Cosine', algorithm 'Brute' & 'Euclidian' distance. And call fit() method on it.
    • call method kneighbors() passing random choice of movie & number of neighbours needed.
    • The above method returns 2 output params. Distances & indice
  • Book recommendation purely on #ratings using Pearson Correlation
    • Each book take avg rating & #ratingcount
    • To ensure statistical significance, Users with less than 200 ratings(across books), and books with 100 ratings are excluded.
    • Create a pivot with user id & book id 
    • Correlate the required book with the pivot. 
  • Book recommendation using Collaborative Filtering(KNN)
    • Each book take avg rating & #ratingcount
    • To ensure statistical significance, Users with less than 200 ratings(across books), and books with 100 ratings are excluded.
    • Create a pivot with user id & book id 
    • Import NearestNeighbours package(This is unsupervised ML)
    • Create variable by initiating NearestNeighbours object passing parameters metric 'Cosine', algorithm 'Brute' & 'Euclidian' distance. And call fit() method on it.
    • call method kneighbors() passing random choice of book & number of neighbours needed.
    • The above method returns 2 output params. Distances & indice
  • Content based Recommendation system

Comments

Post a Comment

Popular posts from this blog

Statistics in Machine Learning

Note : In life cycle of data science more then 60% of time goes into data analysis like feature engineering, feature selection.  Feature engineering means Cleaning data, handle missing values, unbalanced data, category features.   Python packages Pandas - read dataset read_csv, head, isnull, getdummies, drop, concat, Numpy - work with arrays matplotlib.pyplot - for visualization Seaborn - for visualization heatmap, countplot, boxplot, Handling Categorical features one hot encoding for nominal variables label encoding for ordinal variables Ways for finding Outliers Scatter plot Box plot  z-score IQR Correlation :  Strength of association between two variables  This is both ways  A and B = B and A Regression If one of the variable is dependent & other is independent variable Regression equation = Average value of 'y' is a function of x R Square Significance of F & P values     Covariance(cov) Quantify relationship between features, rand...
Read more

Cluster Analysis

Clusters Homogenous within Heterogeneous across Based on field of interest There is no Objective function No dependent variable Popularly called subjective segmentation Segmentation develops on its own based on values of the input variables This is called Unsupervised learning How is Cluster different from decision tree techniques Decision tree technique requires a clearly defined objective funtion this is based on identifying independent and dependent variables. Why clustering Find groups which has elements similar to other groups elements of different groups are dissimilar to each other Generate treatment for specific group  Examples/Business Scenarios In Marketing Leading edge shoppers - looking for technological new products Big brand shoppers Seasonal shoppers Value conscious shoppers Fashion consumer groups Group driven by trends Group interested in Fashion Group purchasing when necessary Cluster analysis as part of business strategy...
Read more