Once Bill Gates said, “640K is enough for anyone”.
However, recent studies on data show that enough is not enough. In fact, it is startling to know while 90% of all the data in the world has been generated over the last two years; it is 2.5 quintillion bytes of data which is generated everyday. So, the big data is getting bigger every day.
Now, the question comes what happens to this humongous unstructured data-Big data. Is it collected and warehoused? Or accessed and analyzed?
The answer lies in the term Big data. The concept is not new because it has been there since 2001. According to definition, Big Data is the- High volume (amount of data), high velocity (speed of collecting and generating data) and high variety information (mostly unstructured data) that requires new forms of processing to enable enhanced decision making, analysis and process optimization.
Companies have been collecting and generating large amounts of data for decades. According to Eric Schmidt, five Exabytes of data is being generated every day by Google. Facebook stores, accesses, and analyzes 30+Petabytes of user generated data with includes audios, videos and pictures. You Tube users upload 48 hours of new video every minute of the day. Every minute of the day 571 new websites are created. According to twitter’s own research in early 2012, it has been observed that about 175 million tweets are generated every day, and more than 465 million accounts are created .So, Google has 20PB ,facebook has 2.5 PB, and e-bay has 6.5 PB of user data to process each day. The speed with which the data is multiplying and growing exponentially it becomes essential for the industry which is driven by data to use it for meaningful analysis and better business decisions so as to create new dimensions to the growth in economy.
One example will illustrate it better. In US 2012 Elections, Big data analysis had been used. Obama’s campaign employed dozens of “data crunchers” that interpreted data, collected and gathered over two years. Needless to say, this helped the campaigners to envision performance, behavior and outcome, and thus they were able to change the content of the campaign for driving the better results. This study shows how collection of data, analysis, and implementation of big data helped campaign’s productivity and success. Another interesting example is of One Trip of flight from New Delhi to London generates which generates about 10 terabytes of data .So, understanding the mechanics of the Big data, we now need to manage this growing data differently so that it can be accessed, captured, and analyzed efficiently for predictions ,trends and profits. Not surprisingly, Big data analysis can be used in future applications like e-traffic control, telecom, fraud and risk management, retail, finance and healthcare etc.
Big data Analysis is the process of examining large amount of variety to uncover hidden patterns, unknown coordination and other useful information. Such information can be used as advantage over rival organization which results in business benefits like more efficacious marketing and elevated revenue. Open source software tools and platforms like Hadoop, HPCC , Map Reduce ,and Rapid Miner are rapidly emerging and evolving to address analytics opportunities. Many companies are emerging with smart tools for Bigdata Analysis - Exalytics (Oracle), Netezza(IBM), Terradata Warehouse Appliance.
So if you don’t have a Chief Analytical Officer, then have one. Next big thing to watch is analyzing data in real time - capturing and processing data in seconds and milliseconds from multiple sources.
School of Engineering