Why can’t we use databases with lots of disks to do large-scale batch analysis?

RDBMS

Why can’t we use databases with lots of disks to do large-scale batch analysis? Why is

Map Reduce needed?

Solution: RDBMS can’t scalable, Hadoop can easily scalable by adding no. of systems.

RDBMS can store only structured data, rather Hadoop can store any Data.(structured/Semi Structured/Un Structured).

	Traditional RDBMS	Map Reduce
Data size	Gigabytes	Petabytes
Access	Interactive and batch	Batch
Updates	Read and write many times	Write once, read many times
Structure	Static schema	Dynamic schema
Integrity	High	Low
Scaling	Nonlinear	Linear

Grid Computing

The High Performance Computing (HPC) and Grid Computing communities have been doing large-scale data processing for years, using such APIs as Message Passing Interface (MPI). Broadly, the approach in HPC is to distribute the work across a cluster

Of machines, which access a shared file system, hosted by a SAN. This works well for predominantly compute-intensive jobs, but becomes a problem when nodes need to access larger data volumes (hundreds of gigabytes, the point at which Map Reduce really

starts to shine), since the network bandwidth is the bottleneck and compute nodes become idle.

Volunteer Computing

                     Volunteer computing projects work by breaking the problem they are trying to solve into chunks called work units, which are sent to computers around the world to be analyzed. For example, a SETI@home work unit is about 0.35 MB of radio telescope data, and takes hours or days to analyze on a typical home computer. When the analysis

is completed, the results are sent back to the server, and the client gets another work unit. As a precaution to combat cheating, each work unit is sent to three different machines and needs at least two results to agree to be accepted. Although SETI@home may be superficially similar to Map Reduce (breaking a problem into independent pieces to be worked on in parallel), there are some significant differences. The SETI@home problem is very CPU-intensive, which makes it suitable for running on hundreds of thousands of computers across the world,since the time to transfer the work unit is dwarfed by the time to run the computation on it. Volunteers are donating CPU cycles, not bandwidth.