Data Storage and Analysis

Data!

We live in the data age. It’s not easy to measure the total volume of data stored electronically, but an IDC estimate put the size of the “digital universe” at 0.18 zettabytes in 2006, and is forecasting a tenfold growth by 2011 to 1.8 zettabytes.1 A zettabyte is 1021 bytes, or equivalently one thousand exabytes, one million petabytes, or one billion terabytes.

Example:  Take photos, for example. My wife’s grandfather was an avid photographer, and took photographs throughout his adult life. His entire corpus of medium format, slide, and 35mm film, when scanned in at high-resolution, occupies around 10 gigabytes.

Data Storage and Analysis

The problem is simple: while the storage capacities of hard drives have increased massively over the years, access speeds—the rate at which data can be read from drives—have not kept up. One typical drive from 1990 could store 1,370 MB of data and had a transfer speed of 4.4 MB/s, so you could read all the data from a full drive in around five minutes. Over 20 years later, one terabyte drives are the norm, but the transfer speed is around 100 MB/s, so it takes more than two and a half hours to read all the data off the disk.

Solution:

This is a long time to read all data on a single drive—and writing is even slower. The obvious way to reduce the time is to read from multiple disks at once. Imagine if we had 100 drives, each holding one hundredth of the data. Working in parallel, we could read the data in less than two minutes.

The first problem to solve is hardware failure: as soon as you start using many pieces

of hardware, the chance that one will fail is fairly high. A common way of avoiding data

loss is through replication: redundant copies of the data are kept by the system so that

in the event of failure, there is another copy available.

The second problem is that most analysis tasks need to be able to combine the data in

some way; data read from one disk may need to be combined with the data from any

of the other 99 disks. Various distributed systems allow data to be combined from

multiple sources, but doing this correctly is notoriously challenging.

This, in a nutshell, is what Hadoop provides: a reliable shared storage and analysis

system. Thestorage is provided by HDFS and analysis by Map Reduce.

Big data spans three dimensions: Volume, Velocity and Variety

Volume: Enterprises are awash with ever-growing data of all types, easily amassing terabytes—even petabytes—of information.

–Turn 12 terabytes of Tweets created each day into improved product sentiment analysis
–Convert 350 billion annual meter readings to better predict power consumption –Turn 12 terabytes of Tweets created each day into improved product sentiment analysis
–Convert 350 billion annual meter readings to better predict power consumption 

  Velocity: Sometimes 2 minutes is too late. For time-sensitive processes such as catching fraud, big data must be used as it streams into your enterprise in order to maximize its value.

–Scrutinize 5 million trade events created each day to identify potential fraud.
–Analyze 500 million daily call detail records in real-time to predict customer churn faster.

The latest I have heard is 10 nano seconds delay is too much.

Variety: Bigdata is any type of data - structured and unstructured data such as text, sensor data, audio, video, click streams, log files and more. New insights are found when analyzing these data types together.–Monitor 100’s of live video feeds from surveillance cameras to target points of interest

Exploit the 80% data growth in images, video and documents to improve customer.