Data!
We live in the data age. It’s not easy to measure
the total volume of data stored electronically, but an IDC estimate put the
size of the “digital universe” at 0.18 zettabytes in 2006, and is forecasting a
tenfold growth by 2011 to 1.8 zettabytes.1 A zettabyte is 1021 bytes, or
equivalently one thousand exabytes, one million petabytes, or one billion terabytes.
Example: Take photos, for example. My wife’s grandfather was an avid photographer,
and took photographs throughout his adult life. His entire corpus of medium
format, slide, and 35mm film, when scanned in at high-resolution, occupies around
10 gigabytes.
The problem is simple: while the storage capacities
of hard drives have increased massively over the years, access speeds—the rate
at which data can be read from drives—have not kept up. One typical drive from
1990 could store 1,370 MB of data and had a transfer speed of 4.4 MB/s, so you
could read all the data from a full drive in around five minutes. Over 20 years
later, one terabyte drives are the norm, but the transfer speed is around 100
MB/s, so it takes more than two and a half hours to read all the data off the
disk.
Solution:
This is a long time to read all data on a single
drive—and writing is even slower. The obvious way to reduce the time is to read
from multiple disks at once. Imagine if we had 100 drives, each holding one
hundredth of the data. Working in parallel, we could read the data in less than
two minutes.
The first problem to solve is hardware failure: as
soon as you start using many pieces
of hardware, the chance that one will fail is
fairly high. A common way of avoiding data
loss is through replication: redundant copies of
the data are kept by the system so that
in the event of failure, there is another copy
available.
The second problem is that most analysis tasks need
to be able to combine the data in
some way; data read from one disk may need to be
combined with the data from any
of the other 99 disks. Various distributed systems
allow data to be combined from
multiple sources, but doing this correctly is
notoriously challenging.
This, in a nutshell, is what Hadoop provides: a
reliable shared storage and analysis
