Hadoop - a Big Data Sneak Peek

What is Hadoop ?
    According to the Apache website Hadoop. It is a software library framework that allows distributed processing of large datasets across clusters of computers using simple programming models.

So How's Hadoop different than regular DBMS ?
Current DBMS can store large datasets in clustered settings but what sets Hadoop apart than regular DBMS systems ?(e.g. Oracle).  Aren't we storing large datasets in the database system and can access them using simple programming models ? The answer to that is BIGDATA. Right now as software systems mature and more complex ones are created organizations are having a hard time storing, managing and analyze the data that they have. So what differentiates BIGDATA from just data.
 The 3 V's 
 Velocity
       - Velocity is the rate at which how an organizations data grows. Data grows really fast. For example Google processes about 24 petabytes per day (ACM White Paper - MapReduce)
 Variety
        - Data comes from all shape and sizes. Most organizations have data scattered all over the place and you can not force all this data into a single schema. This is specially true when organizations merge (believe me 30 year old systems are still out there holding important data in a format that most college graduate would not recognize).
  and specially Volume
        - Don't think 30,000 employee records, think humongous like in the range of Data's memory capacity (Data from Star Trek not so far fetched). Petabytes of data !

How do you process such kind of Data ?
   That is the problem space that Hadoop is trying to solve. Hadoop again is a reliable, fault tolerant, high performance distributed parallel programming framework for large scale data written in Java.

Two Parts of Hadoop
    1. HDFS (Hadoop File System)
        This is a distributed file system that is fault tolerant (data is replicated) and runs on inexpensive commodity hardware (hardware you can buy from Best Buy)
    2. MapReduce
          This is the programming model that you can use to process and analyze data in Hadoop. This is the same programming model that Google uses.

Now that you have an idea what Hadoop is then you should have an idea of what it is not for.  Here are some suggestions:
Don't store small files - hadoop is not efficient storing small files. Think big!
Don't use it for online applications - HDFS is not highly available for online applications

In my next post, I will try and discuss the parts of HDFS and touch a little bit on a MapReduce sample.


Comments

Post a Comment

Popular posts from this blog

OAuth 1.0a Request Signing and Verification - HMAC-SHA1 - HMAC-SHA256

Image
This is a security project that I did this Fall semester. I was interested in why the OAuth1.0a specification did not require you to sign a request using RSA-SHA1 or any RSA algorithm. The first issue that I found is where would you store your private keys in a javascript client? - as a javascript encrypted file ? from the browser ?  - that will be a security hole if you allow javascript to get private keys. These are some of the questions that got me asking and which I cannot find answers to for now. OAuth 1.0a Signature Signing and Verification using JavaScript Implementation Background The OAuth specification tries to define an API authentication method that allows consumer websites or applications to access protected API Resources from a Service provider. The advantage that OAuth provides is that the user of the website or application is not required to disclose their credentials to their service provider. Another advantage of using OAuth to authenticate is that it does not
Read more

Spark DataFrame - Array[ByteBuffer] - IllegalAurmentException

IllegalArgumentException - ByteBuffer - Spark DataFrame I was processing a several million documents (~ 20 million) in which we need to extract the NLP features using NLP4J, OpenNLP, and WordNet. The combination of the three NL features blows up each record to 11 times its original size. We are using all three because we do not know yet what feature sets will be helpful to us. The original dataset is in parquet files in HDFS (16 partitions). I thought that was convenient just use withColumn and pass a UDF (User Defined Function) on the column where it needs those features. withColumn adds the calculated column back to the DataFrame. So I created the spark job (I am on Spark 1.5.2-cdh5.5.2)for the above, and things started to get nasty. I am blowing up the ByteBuffer array on the in-memory columnar storage. This is the exception that I am getting. There seems to be no reference in my code in this stack trace. java.lang.IllegalArgumentException at java.nio.ByteB
Read more

Gensim Doc2Vec on Spark - a quest to get the right Vector

Ever since I joined the R & D group we have been doing a lot of cool things, like trying IBM Watson (see previous blog entry). Now we are doing a lot of Natural language processing.  We wanted to compare the similarity of two documents. There is this excellent project Gensim ( doc2vec ) that easily allow to you to translate large blocks of text to a fixed length feature vector to make comparisons. Here is the link to the  original paper  from some people from Google that explains the approach. In essence, they wanted to find a representation that will overcome the weaknesses of the bag of words model. The doc2vec approach proves to be a reliable approach to comparing similarities of documents because it takes into consideration the semantics and the order of the word in context. So with that, we wanted to use it for a corpus of 26 million documents. Calculating the doc2vec for 26 million documents is not a small task, so we need to process it in Spark. The problem is that there
Read more