Watson - The mystery after jeopardy

We have been deep diving in cognitive computing. One of the best platforms that a business can leverage to hit the ground on cognitive computing is IBM Watson. Watson has a lot of capabilities especially with the acquisition of Alchemy's API as well. (Alchemy Acquisition - IBM). You get a language translator, language classifier, retrieve and rank, text to speech, tone analyzer, and a lot more. It is just a matter of how these capabilities can be integrated to your business use cases. As part of "the answer" company we have a tremendous and diverse use case for searching - and giving you answers in a way that makes sense, relevant and make a user decide better is at the heart of what makes us "the answer" company. I was a part of the team given the freedom to explore IBM Watson (no matter what the cost). So we have tried the different APIs in a span of a few weeks. Of course, we have to take a look at the Watson's retrieve and rank (IBM Watson;s Retrieve and Rank). Learning to rank have been researched extensively(Learn to Rank) and we have been trying out a few things ourselves so we are quite versed on the retrieving and ranking. Here are the three things that I have observed with Watson's Retrieve and Rank


1. Data! Data! Data! I can't make brick without clay (Can't make bricks without clay)

   - Well before you can train IBM's retrieve and rank. The data that you will need to use for your ground truth should meet a minimum quality standard which is well described in the documentation here (Retrieve and Rank documentation). If you have a lot of data, meeting the requirements in the documentation is easy enough. Once, you have met the data requirements you will probably need a translation from the original labels in your data to the relevance label that retrieves and ranks needs. This can be a subject of misinterpretation especially if you are evaluating performance. Evaluating performance might be tricky again because you might need to translate the relevance label to yet another mapping - especially if you are using a third party library like TREC Eval (TREC Eval). I think that IBM should provide all those testing capabilities in Bluemix.

2. Everyone loves a mystery to be solved

  - One of the things that we wanted to uncover was how can we turn the knobs/levers on how to the re-rank portion of retrieve and rank behave to better suit our needs. The rank portion is IBM's secret sauce - there is no quick turning of knobs. It does some magic in mysterious ways, that no one using the API will ever know (unless you work for IBM =). The way to add a little bit of spice to IBM's secret sauce is to use a proxy, which is described here (Retrieve and Rank - Proxy). I think the disconnect here is that our team has the expertise in turning the knobs especially in machine learning and language processing. Other businesses might not have this expertise. So having the magic happen in the background and have IBM solve it is good enough for most organizations.

3. Watson will never have the same deductive skills as Sherlock

 - Since our company have been doing language processing for a long time. We discovered that we have been doing a little more advanced things than what IBM is doing in the retrieve and rank space. This is good to know because we know we have the capability to address the businesses needs to become "the answer" company.


So there is no mystery in IBM's retrieve and rank API. We know that we are not eager to use it. Right, now it doesn't suit our needs. Maybe if IBM will add more capabilities and make it easier for us to tweak - so that we can have a little bit of control on how it ranks - we might use it. The biggest hindrance I think to its adoption in our organization is probably the cost of running the API in Bluemix. We probably don't want to be charged a fee every time a user does a search. We racked up a few thousand dollars of the bill while exploring in a span of two weeks (probably because we have a lot of data). Hit some limits on the API. One more thing, you have to transport your data in IBM's server in order to use retrieve and rank - well that might not suit organizations whose core business is the data. Especially with legislation like this (Personal Data for Russian Citizens), you might want to control where you transport your data.

Comments

Post a Comment

Popular posts from this blog

OAuth 1.0a Request Signing and Verification - HMAC-SHA1 - HMAC-SHA256

Image
This is a security project that I did this Fall semester. I was interested in why the OAuth1.0a specification did not require you to sign a request using RSA-SHA1 or any RSA algorithm. The first issue that I found is where would you store your private keys in a javascript client? - as a javascript encrypted file ? from the browser ?  - that will be a security hole if you allow javascript to get private keys. These are some of the questions that got me asking and which I cannot find answers to for now. OAuth 1.0a Signature Signing and Verification using JavaScript Implementation Background The OAuth specification tries to define an API authentication method that allows consumer websites or applications to access protected API Resources from a Service provider. The advantage that OAuth provides is that the user of the website or application is not required to disclose their credentials to their service provider. Another advantage of using OAuth to authenticate is that it does not
Read more

Spark DataFrame - Array[ByteBuffer] - IllegalAurmentException

IllegalArgumentException - ByteBuffer - Spark DataFrame I was processing a several million documents (~ 20 million) in which we need to extract the NLP features using NLP4J, OpenNLP, and WordNet. The combination of the three NL features blows up each record to 11 times its original size. We are using all three because we do not know yet what feature sets will be helpful to us. The original dataset is in parquet files in HDFS (16 partitions). I thought that was convenient just use withColumn and pass a UDF (User Defined Function) on the column where it needs those features. withColumn adds the calculated column back to the DataFrame. So I created the spark job (I am on Spark 1.5.2-cdh5.5.2)for the above, and things started to get nasty. I am blowing up the ByteBuffer array on the in-memory columnar storage. This is the exception that I am getting. There seems to be no reference in my code in this stack trace. java.lang.IllegalArgumentException at java.nio.ByteB
Read more

Gensim Doc2Vec on Spark - a quest to get the right Vector

Ever since I joined the R & D group we have been doing a lot of cool things, like trying IBM Watson (see previous blog entry). Now we are doing a lot of Natural language processing.  We wanted to compare the similarity of two documents. There is this excellent project Gensim ( doc2vec ) that easily allow to you to translate large blocks of text to a fixed length feature vector to make comparisons. Here is the link to the  original paper  from some people from Google that explains the approach. In essence, they wanted to find a representation that will overcome the weaknesses of the bag of words model. The doc2vec approach proves to be a reliable approach to comparing similarities of documents because it takes into consideration the semantics and the order of the word in context. So with that, we wanted to use it for a corpus of 26 million documents. Calculating the doc2vec for 26 million documents is not a small task, so we need to process it in Spark. The problem is that there
Read more