Why can’t we use databases with lots of disks to do large-scale batch analysis?

RDBMS

Why can’t we use databases with lots of disks to do large-scale batch analysis? Why is

Map Reduce needed?

Solution: RDBMS can’t scalable, Hadoop can easily scalable by adding no. of systems.

RDBMS can store only structured data, rather Hadoop can store any Data.(structured/Semi Structured/Un Structured).

	Traditional RDBMS	Map Reduce
Data size	Gigabytes	Petabytes
Access	Interactive and batch	Batch
Updates	Read and write many times	Write once, read many times
Structure	Static schema	Dynamic schema
Integrity	High	Low
Scaling	Nonlinear	Linear

Grid Computing

The High Performance Computing (HPC) and Grid Computing communities have been doing large-scale data processing for years, using such APIs as Message Passing Interface (MPI). Broadly, the approach in HPC is to distribute the work across a cluster

Of machines, which access a shared file system, hosted by a SAN. This works well for predominantly compute-intensive jobs, but becomes a problem when nodes need to access larger data volumes (hundreds of gigabytes, the point at which Map Reduce really

starts to shine), since the network bandwidth is the bottleneck and compute nodes become idle.

Volunteer Computing

                     Volunteer computing projects work by breaking the problem they are trying to solve into chunks called work units, which are sent to computers around the world to be analyzed. For example, a SETI@home work unit is about 0.35 MB of radio telescope data, and takes hours or days to analyze on a typical home computer. When the analysis

is completed, the results are sent back to the server, and the client gets another work unit. As a precaution to combat cheating, each work unit is sent to three different machines and needs at least two results to agree to be accepted. Although SETI@home may be superficially similar to Map Reduce (breaking a problem into independent pieces to be worked on in parallel), there are some significant differences. The SETI@home problem is very CPU-intensive, which makes it suitable for running on hundreds of thousands of computers across the world,since the time to transfer the work unit is dwarfed by the time to run the computation on it. Volunteers are donating CPU cycles, not bandwidth.

Data Storage and Analysis

Data!

We live in the data age. It’s not easy to measure the total volume of data stored electronically, but an IDC estimate put the size of the “digital universe” at 0.18 zettabytes in 2006, and is forecasting a tenfold growth by 2011 to 1.8 zettabytes.1 A zettabyte is 1021 bytes, or equivalently one thousand exabytes, one million petabytes, or one billion terabytes.

Example:  Take photos, for example. My wife’s grandfather was an avid photographer, and took photographs throughout his adult life. His entire corpus of medium format, slide, and 35mm film, when scanned in at high-resolution, occupies around 10 gigabytes.

Data Storage and Analysis

The problem is simple: while the storage capacities of hard drives have increased massively over the years, access speeds—the rate at which data can be read from drives—have not kept up. One typical drive from 1990 could store 1,370 MB of data and had a transfer speed of 4.4 MB/s, so you could read all the data from a full drive in around five minutes. Over 20 years later, one terabyte drives are the norm, but the transfer speed is around 100 MB/s, so it takes more than two and a half hours to read all the data off the disk.

Solution:

This is a long time to read all data on a single drive—and writing is even slower. The obvious way to reduce the time is to read from multiple disks at once. Imagine if we had 100 drives, each holding one hundredth of the data. Working in parallel, we could read the data in less than two minutes.

The first problem to solve is hardware failure: as soon as you start using many pieces

of hardware, the chance that one will fail is fairly high. A common way of avoiding data

loss is through replication: redundant copies of the data are kept by the system so that

in the event of failure, there is another copy available.

The second problem is that most analysis tasks need to be able to combine the data in

some way; data read from one disk may need to be combined with the data from any

of the other 99 disks. Various distributed systems allow data to be combined from

multiple sources, but doing this correctly is notoriously challenging.

This, in a nutshell, is what Hadoop provides: a reliable shared storage and analysis

system. Thestorage is provided by HDFS and analysis by Map Reduce.

What Do You Do with Data

Lets take an example:

                 Ø From application developers to video streamers, organizations of all sizes face the challenge of capturing, searching, analyzing, and leveraging as much as terabytes of data per second—too much for the constraints of traditional system capabilities and database management tools.”

Why Big Data ?

§  Key enablers for the appearance and growth of ‘Big-Data’ are:

+     Increase in storage capabilities

+     Increase in processing power

+     Availability of data

When dealing with Big Data is hard:   

§  When the operations on data are complex:

–     Eg. Simple counting is not a complex problem.

–     Modeling and reasoning with data of different kinds can get extremely complex

§  Good news with big-data:

–     Often, because of the vast amount of data, modeling techniques can get simpler (e.g., smart counting can replace complex model-based analytics)…

–     …as long as we deal with the scale.

Types of tools typically used in Big Data Scenario

§  Where is the processing hosted?

–     Distributed server/cloud

§  Where data is stored?

–     Distributed Storage (eg: Amazon s3)

§  Where is the programming  model?

–     Distributed processing (Map Reduce)

§  How data is stored and indexed?

–     High performance schema free database

§  What operations are performed on the data?

         –     Analytic/Semantic Processing (Eg. RDF/OWL)

Big data spans three dimensions: Volume, Velocity and Variety

Volume: Enterprises are awash with ever-growing data of all types, easily amassing terabytes—even petabytes—of information.

–Turn 12 terabytes of Tweets created each day into improved product sentiment analysis
–Convert 350 billion annual meter readings to better predict power consumption –Turn 12 terabytes of Tweets created each day into improved product sentiment analysis
–Convert 350 billion annual meter readings to better predict power consumption 

  Velocity: Sometimes 2 minutes is too late. For time-sensitive processes such as catching fraud, big data must be used as it streams into your enterprise in order to maximize its value.

–Scrutinize 5 million trade events created each day to identify potential fraud.
–Analyze 500 million daily call detail records in real-time to predict customer churn faster.

The latest I have heard is 10 nano seconds delay is too much.

Variety: Bigdata is any type of data - structured and unstructured data such as text, sensor data, audio, video, click streams, log files and more. New insights are found when analyzing these data types together.–Monitor 100’s of live video feeds from surveillance cameras to target points of interest

Exploit the 80% data growth in images, video and documents to improve customer. 

Introduction To Hadoop Ecosystem and Big Data - For Beginners

Every day we create more than 2.5 quintillion bytes of data- so much that 90% of the data in the world today has been created in the last two years alone. The increase in volume of digital information is alnost tenfold every 5 years and by 2020, it is predicted to reach a threshold of 40 zettabytes (ZB). Most of the digital information is derived from multiple sources including social media, internet, blogs, sensor networks, and much more

                          

 This data is "Big Data."

                           
Huge Amount of Data:

There is huge amount of data in the world

Ø From the beginning till 2003 almost we have create 5 billion gigabytes(exabytes) of data.

Ø In the year 2011 we have created same amount of data with in every 2 days

Ø In the year 2013 we have created same amount of data with in a time of 10 minutes