When
a dataset outgrows the storage capacity of a single physical machine, it
becomes necessary to partition it across a number of separate machines.
Filesystems that manage the storage across a network of machines are called distributed filesystems.
Apache Hadoop is an Open source
Distributed system with fault tolerance Mechanism.
Hadoop comes with a distributed filesystem called
HDFS, which stands for Hadoop
Distributed Filesystem.
Cluster: All the
nodes(systems) should be under common dedicated network. Minimum size of
Cluster is “1”.
Hadoop
Cluster:
The Nodes should be appended with both HDFS and MAPREDUCE.
VPN: Virtual
private Network.
The Design of HDFS
HDFS is a filesystem designed
for storing very large files with streaming data access patterns, running on
clusters of commodity hardware. Let’s examine this statement in more detail:
Very large files
“Very large” in this context means files that are hundreds of megabytes,
gigabytes, or terabytes in size. There are Hadoop clusters running today that
store petabytes of data.
Streaming data access
HDFS
is built around the idea that the most efficient data processing pattern is a write-once,
read-many-times pattern. A dataset is typically generated or copied from
source, and then various analyses are performed on that dataset over time. Each
analysis will involve a large proportion, if not all, of the dataset, so the
time to read the whole dataset is more important than the latency in reading
the first record.
Commodity hardware
Hadoop doesn’t require
expensive, highly reliable hardware to run on. It’s designed to run on clusters
of commodity hardware (commonly available hardware available from multiple
vendors3) for which the chance of node failure across the cluster is high, at
least for large clusters. HDFS is designed to carry on working without a noticeable
interruption to the user in the face of such failure.
Low-latency data access
Applications that require
low-latency access to data, in the tens of milliseconds range, will not work
well with HDFS. Remember, HDFS is optimized for delivering a high throughput of
data, and this may be at the expense of latency. HBase is currently a better
choice for low-latency access.
Lots of small files
Since the namenode holds
filesystem metadata in memory, the limit to the number of files in a filesystem
is governed by the amount of memory on the namenode. As a rule of thumb, each
file, directory, and block takes about 150 bytes. So, for example, if you had
one million files, each taking one block, you would need at least 300 MB of
memory. While storing millions of files is feasible, billions is beyond the
capability of current hardware.
Write Once read many times
Files in HDFS may
be written to by a single writer. Writes are always made at the end of the
file. There is no support for multiple writers, or for modifications at arbitrary
offsets in the file. (These might be supported in the future, but they are likely
to be relatively inefficient.)
Moving Business logic near to the Data is easy than
moving the Data to the Business logic
That means all the
computations are moved to the Data rather than moving data near to the Business
logic.
