Recommendations - Hey user, I think you might like this!

Would you likely read this ?

Recommendation has been part of most online systems for quite some time now. Amazon, Target, and Netflix have recommendation algorithms that gets you to look at more items either to buy or watch. The basic premise of recommendation is to use collected historical data for a user in order to determine items or products that might be valuable to a user. Determining the recommended item is based on a strategic plan of what the goal is for recommending. It is usually for a user to purchase additional products or something that will significantly help the user. The "strategic plan" is usually something based on a business goal to push (recommend) to the user. Pushing an item for recommendation without real significance and context is not really a good plan because users will ignore such recommendations with high probability and will also alienate users (kind of like the web sites in facebook that are full of advertisements that does not really make sense. Like answers.com - I really hate this!). High quality recommendations for the users of a web site usually incorporates helping the user discover something that is significant based on the current usage context (kind of like this - Target really being helpful !) . Significant recommendations are determined by a number of statistical techniques being employed using the data of the user.

Having been really involve implementing recommendations using statistical techniques, there was a lot of collaborating with our R&D team in order to come out with the best recommendation algorithms. (I have to dig deep to my statistics subject in college.) There are a lot of recommendation logic out there. After working on this subject, I have observed different techniques in some of the recommendations in the websites i use.

Chi-Square ("kye"- square)

At the heart of some of the recommendation algorithms that i have implemented is the Chi-square test. Basically, we want to quantify a users preference for a certain kind of item based on historical usage. In order to do that we will need how many and what kind of items they have looked up in the past. This technique will be good if you have a large user base and a significant amount of historical user data.

There are specialized variations of this statistical technique. Usually a variation on the data points collected for the user like "user clicks", "user purchases", and "user views".  Statistical analysis of user data helps a lot in providing significant recommendations.

Recommendations may not always come from usage data. It might also just be a straight out business rule. As an example, if a grocery store website is trying to push its inventory of milk, it might just always recommend milk every time (but this is not really helping the user).


Where to now ?

Statistical techniques will have its limits based on the data on hand. It is not really deep in the sense that the recommendations are based on historical data which is not always a good basis in predicting the future behavior of users. In order to take recommendations further it should employ a cognitive architecture (artificial intelligence) that can completely react or interact with users. Recommendations based on real time interaction will have a higher chance of being significant and current.

Talking about AI. Try my twitter bot @petabyteflop. You can ask any kind of question. This is a study on how customer service can be automated in twitter.






Comments

Post a Comment

Popular posts from this blog

OAuth 1.0a Request Signing and Verification - HMAC-SHA1 - HMAC-SHA256

Image
This is a security project that I did this Fall semester. I was interested in why the OAuth1.0a specification did not require you to sign a request using RSA-SHA1 or any RSA algorithm. The first issue that I found is where would you store your private keys in a javascript client? - as a javascript encrypted file ? from the browser ?  - that will be a security hole if you allow javascript to get private keys. These are some of the questions that got me asking and which I cannot find answers to for now. OAuth 1.0a Signature Signing and Verification using JavaScript Implementation Background The OAuth specification tries to define an API authentication method that allows consumer websites or applications to access protected API Resources from a Service provider. The advantage that OAuth provides is that the user of the website or application is not required to disclose their credentials to their service provider. Another advantage of using OAuth to authenticate is that it does not
Read more

Spark DataFrame - Array[ByteBuffer] - IllegalAurmentException

IllegalArgumentException - ByteBuffer - Spark DataFrame I was processing a several million documents (~ 20 million) in which we need to extract the NLP features using NLP4J, OpenNLP, and WordNet. The combination of the three NL features blows up each record to 11 times its original size. We are using all three because we do not know yet what feature sets will be helpful to us. The original dataset is in parquet files in HDFS (16 partitions). I thought that was convenient just use withColumn and pass a UDF (User Defined Function) on the column where it needs those features. withColumn adds the calculated column back to the DataFrame. So I created the spark job (I am on Spark 1.5.2-cdh5.5.2)for the above, and things started to get nasty. I am blowing up the ByteBuffer array on the in-memory columnar storage. This is the exception that I am getting. There seems to be no reference in my code in this stack trace. java.lang.IllegalArgumentException at java.nio.ByteB
Read more

Gensim Doc2Vec on Spark - a quest to get the right Vector

Ever since I joined the R & D group we have been doing a lot of cool things, like trying IBM Watson (see previous blog entry). Now we are doing a lot of Natural language processing.  We wanted to compare the similarity of two documents. There is this excellent project Gensim ( doc2vec ) that easily allow to you to translate large blocks of text to a fixed length feature vector to make comparisons. Here is the link to the  original paper  from some people from Google that explains the approach. In essence, they wanted to find a representation that will overcome the weaknesses of the bag of words model. The doc2vec approach proves to be a reliable approach to comparing similarities of documents because it takes into consideration the semantics and the order of the word in context. So with that, we wanted to use it for a corpus of 26 million documents. Calculating the doc2vec for 26 million documents is not a small task, so we need to process it in Spark. The problem is that there
Read more