2016 - Movies Data Analysis - Linear Regression Modelling

Popular posts from this blog

Attention not needed!

Image
Attention, Attention For as long as we have been using Transformer models, there has been a question about how to put it in production because it needs a lot of computing power. Google took about a year from when BERT (Bidirectional Encoder Representations from Transformers) was introduced to being incorporated into their search algorithm. It is not just the advancement in understanding searches using BERT; they also needed new hardware. “But it’s not just advancements in software that can make this possible: we needed new hardware too. Some of the models we can build with BERT are so complex that they push the limits of what we can do using traditional hardware, so for the first time, we’re using the latest  Cloud TPUs  to serve search results and get you more relevant information quickly. “( https://blog.google/products/search/search-language-understanding-bert/ ) One of the complexities in BERT is the ‘self-attention’ modules, these are neural network layers that compu...
Read more

Spark DataFrame - Array[ByteBuffer] - IllegalAurmentException

IllegalArgumentException - ByteBuffer - Spark DataFrame I was processing a several million documents (~ 20 million) in which we need to extract the NLP features using NLP4J, OpenNLP, and WordNet. The combination of the three NL features blows up each record to 11 times its original size. We are using all three because we do not know yet what feature sets will be helpful to us. The original dataset is in parquet files in HDFS (16 partitions). I thought that was convenient just use withColumn and pass a UDF (User Defined Function) on the column where it needs those features. withColumn adds the calculated column back to the DataFrame. So I created the spark job (I am on Spark 1.5.2-cdh5.5.2)for the above, and things started to get nasty. I am blowing up the ByteBuffer array on the in-memory columnar storage. This is the exception that I am getting. There seems to be no reference in my code in this stack trace. java.lang.IllegalArgumentException at java.nio.B...
Read more

Speech to Text - HTML 5

Technology Can Help Recently there was a news in Good Morning America were a deaf cashier is taking orders and customers patiently writing out there orders. Here is a link   Deaf Cashier Well with a computer in every pocket there is technology to help. Here is a quick demo that i have created to use HTML 5 technology to create a speech to text web page. Here is a link to the working page http://petabyte.github.io/textToSpeech.html https://www.youtube.com/watch?v=27SZISZAPEA
Read more