2016 - Movies Data Analysis - Linear Regression Modelling

Popular posts from this blog

ElasticSearch - Sniff ! Sniff!

Note: Below is a brief discussion of the issues i have encountered using Elasticsearch's  Client  behind a VIP. As I had written previously, I had to setup an Elasticsearch cluster. In order to take advantage of balancing the load between the nodes of the cluster. I used a DNS pointed to VIP that does a round-robin load balancing among the nodes. The reason i choose this is because I can add servers to the VIP without changing the Transport Addresses in the Elasticsearch Client java code which uses a TransportClient. Without the VIP I need a coding change and deploy to add the new Transport Address. The cluster would not be really useful if I cannot take away nodes and add nodes to the cluster as I wish without any coding change. The drawback in hiding the cluster behind the VIP is that you don't know which node the java code client is going to get in order to process a search request.  This is causing an issue in the client code.  Since I only have one Tran...
Read more

Gensim Doc2Vec on Spark - a quest to get the right Vector

Ever since I joined the R & D group we have been doing a lot of cool things, like trying IBM Watson (see previous blog entry). Now we are doing a lot of Natural language processing.  We wanted to compare the similarity of two documents. There is this excellent project Gensim ( doc2vec ) that easily allow to you to translate large blocks of text to a fixed length feature vector to make comparisons. Here is the link to the  original paper  from some people from Google that explains the approach. In essence, they wanted to find a representation that will overcome the weaknesses of the bag of words model. The doc2vec approach proves to be a reliable approach to comparing similarities of documents because it takes into consideration the semantics and the order of the word in context. So with that, we wanted to use it for a corpus of 26 million documents. Calculating the doc2vec for 26 million documents is not a small task, so we need to process it in Spark. The problem ...
Read more

Node.js - lightweight and efficient

Using Node.js you can write a web server in as little as six lines of code. Here is a variation for the sample from the Node.js  (Node.js Synopsis) //Imports the http library var = require(http); http.createServer(function(request, response){     response.writeHead(200,{'Content-Type':'text/plain'});     response.end('Hello World\n'); }).listen(8124); console.log('Server running at http://127.0.0.1:8124/'); I was asking myself why would i want a server side javascript (which is the whole idea of Node.js) ? Besides that it uses a non-blocking I/O model (this means that other processing can continue if it does not depend fro an I/O to complete, kind of like an asynchronous call to perform the I/O) . One use case that i can think of is that  I can wrap other services (e.g. other source of data like json/xml)  specially ones that are exposed as web service APIs (e.g. Google APIs).
Read more