Petabyte

APE - Author Publisher Entrepreneur  APE Website I will be reviewing, commenting and sharing my feelings about the book. My plan is to read each section of the book [Author, Publisher, Entrepreneur] and posts my review on this blog. Author - The first seven chapters deals with the realities of what an author faces trying to publish a book.This is the section that help you answer the question "Should i write a book ?". If after reading this section and you answer is still yes, then it gives you a little bit of guidance on how to start things off. [The Art of the Start - Guy Kawasaki]. I liked that the authors pointed out the mountain that you have to face publishing a book the traditional way because it takes away the fantasy that once you are done writing your manuscript you are on your way to selling a bestseller.The authors also pointed out how self-publishing can start a revolution that everybody can publish a book. Everybody can be an author as long as you are wil

Here is the next installment for Hadoop. I hope this gives you an idea of what HDFS and MapReduce in Hadoop is. Please leave a comment if you find this useful. HDFS Architecture The diagram below shows the main components for the Hadoop File System (HDFS). Name Node - This is the central piece of HDFS. The Name Node tracks all the  file system's metadata (e.g. directory tree of all files, where the file is kept in the cluster). It does not store the data itself. The Name Node is the SPOF (Single Point of Failure in HDFS). Here is a link to a more detailed definition -> NameNode Secondary Name Node - optional component of the HDFS. Creates checkpoints of the namespaces Data Node - Stores data in large blocks in the filesystem. Reports the blocks of data it holds to the Name Node. MapReduce MapReduce is a programming model on a distributed processing platform that is scalable. It is actually as three step process. 1. Map - this steps creates a map (key-val

What is Hadoop ?     According to the Apache website  Hadoop . It is a software library framework that allows distributed processing of large datasets across clusters of computers using simple programming models. So How's Hadoop different than regular DBMS ? Current DBMS can store large datasets in clustered settings but what sets Hadoop apart than regular DBMS systems ?(e.g. Oracle).  Aren't we storing large datasets in the database system and can access them using simple programming models ? The answer to that is BIGDATA. Right now as software systems mature and more complex ones are created organizations are having a hard time storing, managing and analyze the data that they have. So what differentiates BIGDATA from just data.  The 3 V's    Velocity        - Velocity is the rate at which how an organizations data grows. Data grows really fast. For example Google processes about 24 petabytes per day ( ACM White Paper - MapReduce )  Variety         - Data comes f

Follow this link -->  HTML 5 Quick Overview - prezi Please leave comment and follow me! Thanks!

I have been working with a lot (when i mean a lot ~50++) of threads. It is necessary for our applications because we deal with a lot (~millions of records). Why is it necessary to multi-thread ? Let's say you have 1,000,000 records to process and each records is processes roughly about 5 seconds (really intense processing e.g. calling web services, writing xml, uploading via ftp). It will take you roughly 57 days, 20 hours, 53 minutes, 20 seconds to process all of the records ( 5,000,000 seconds ). Since you are dependent on other systems (e.g. like web services to process the records) you can't really ask the publisher of the service to speed things up (unless you have are willing to throw in time and effort into the service before you process your records). So, enter multi threading. Multi Threading - enables you to leverage the multiple processors (common to most enterprise server level boxes) to process records in parallel. BUT- multi threaded programs are really hard t

Using Node.js you can write a web server in as little as six lines of code. Here is a variation for the sample from the Node.js  (Node.js Synopsis) //Imports the http library var = require(http); http.createServer(function(request, response){     response.writeHead(200,{'Content-Type':'text/plain'});     response.end('Hello World\n'); }).listen(8124); console.log('Server running at http://127.0.0.1:8124/'); I was asking myself why would i want a server side javascript (which is the whole idea of Node.js) ? Besides that it uses a non-blocking I/O model (this means that other processing can continue if it does not depend fro an I/O to complete, kind of like an asynchronous call to perform the I/O) . One use case that i can think of is that  I can wrap other services (e.g. other source of data like json/xml)  specially ones that are exposed as web service APIs (e.g. Google APIs).

The Future When dealing with concurrent processing, starting Java 1.5 the 'Future' looks bright. In java 1.5 it is now easier to collect results from asynchronous processes by getting the future. Here are the objects that you are going to need to process concurrently and collect results from those concurrent processes: //Please refer the the Java API documentation on how to properly instantiate the objects below //This is just to give an overview on how they are used together and relate to each other. //A ThreadPoolExecutor is basically a pool of threads that you configure that will asynchronously //perform the tasks that you provide it ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(); //A completion service is the object that collects the results of the task that the ThreadPoolExecutor //completes. That is why when you instantiate it you need to pass it a ThreadPoolExecutor. CompletionService completionService = new ExecutorCompletionService (th