DATA NODE

3.  DATA NODE

-  DATANODES are the workhorse of the HADOOP CLUSTER.

-  All the Data resides on top of DN.

-  DN can read or write a HDFS file (Actually the file is broken into blocks and the Name  Node will tell your client which Data Node each block resides in).

- Upon initialization, each of the Data Nodes informs the Name Node of the blocks it is currently storing.

- After this mapping is complete, the Data Nodes continually poll the Name Node to provide information regarding local changes as well as receive instructions to create, move, or delete blocks from the local disk.

- Your client communicates directly with the Data Node daemons to process the local files corresponding to the blocks.  

- Data Node may communicate with other Data Nodes to replicate its data blocks for redundancy. 

- The Data Nodes provide backup store of the blocks and constantly report to the Name Node to keep the metadata current. 

- Backup store of the blocks ensures that if any one Data Node crashes or becomes inaccessible over the network, you will still be able to read the files.

SECONDARY NAMENODE

2.  SECONDARY NAMENODE

-  SECONDARY NAMENODE will be doing housekeeping activities of NAMENODE.

-  SNN cannot replace the NN in any case of failure.

-  SNN is to perform periodic checkpoints. It will periodically downloads the FSIMAGE and EDITLOG file and merge both of them and keep EDITLOG size with in limit.

-  So if the NN fails we can get a copy of Metadata in the Format of FSIMAGE from SNN.

-  SNN will be coping FSIMAGE from the NN in checkpoint manner.

NAME NODE (NN)

1.  NAME NODE

-  NAME NODE will act like an MASTER and it will manages the file system namespace. It                                                  maintains the filesystem tree and the metadata for all the files and the directories in the tree.

-  NN will store the METADATA.

- METADATA is the data about physical location of the blocks & by combining which   blocks the data will be prepared.

-  In any case if the NN failed it is called SPOF (single point of failure).

-  The NN will only communicate with clients.

-  The NN is also having capabilities of storing Data in it.

-  NN is very sensitive, even though failures are very rare.

PERSISTANCE FILE SYSTEM in NN

This filesystem metadata is stored in two different constructs: the FSIMAGE and the EDIT LOG.

Ø  FSIMAGE  

Ø  EDITLOG

                                         The fsimage is a file that represents a point-in-time snapshot of the filesystem’s metadata. However, while the fsimage file format is very efficient to read, it’s unsuitable for making small incremental updates like renaming a single file. Thus, rather than writing a new fsimage every time the namespace is modified, the NameNode instead records the modifying operation in the edit log for durability. This way, if the NameNode crashes, it can restore its state by first loading the fsimage then replaying all the operations (also called edits or transactions) in the edit log to catch up to the most recent state of the namesystem. The edit log comprises a series of files, called edit log segments, that together represent all the namesystem modifications made since the creation of the fsimage.             

CHECK POINT MECHANISUM

                 Check pointing is a process that takes an fsimage and edit log and compacts them into a new fsimage. This way, instead of replaying a potentially unbounded edit log, the NameNode can load the final in-memory state directly from the fsimage. This is a far more efficient operation and reduces NameNode startup time.

HADOOP ARCHITECTURE

In our Hadoop Architecture we have 5 Daemons, they are

1.  NAME NODE (NN)

2.  SECONDARY NAME NODE (SNN)

3.  DATA NODE (DN)

4.  JOB TRACKER (JT)

5.  TASK TRACKER (TT)

- In Cloudera Distribution Hadoop (CDH) Mechanism we have Single NAME NODE with the help of SECONDARY NAME NODE.

-  In MAP R Mechanism we can maintain multiple NAME NODES.

Hadoop Distributed File System Concepts

HDFS Concepts

Blocks

A disk has a block size, which is the minimum amount of data that it can read or write.

Hadoop also has a concept of a block, minimum block size—64 MB by default.

Files in HDFS are broken into block-sized chunks, which are stored as independent units.

Except the last block remaining all blocks are of equal size.

The Blocks cannot be divided or shared between the files.

Hadoop can process large files efficiently instead of small files.

Block size can also be changed with the help of configurations.

                When we install Hadoop, the Hadoop folder contains some configuration files the Ex: path is /Hadoop/conf/hdfs-site.xml   in this file we have to configure the block size.

<configuration>

<property>

<name>dfs.block.size</name>

<value>141213058</value>

</property>

</configuration>

Replication in HADOOP

Replication is nothing but maintaining same copy of file in another node.

-In our Traditional distributed system, if one node is fail we have to wait until it is recovered or backup is done, till then the result cannot be released.

-But in Hadoop we can get the data from another node as soon as the node fails. It will be possible with the help of replication.

Formula for replication is = n-1  where n = no. of slave nodes

Hadoop is always expecting commodity Software and Hardware.

The default replication factor is “3”, we can also configure using configuration file.

/Hadoop/conf/hdfs-site.xml  in this file we have set the property.

<configuration>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

</configuration>

-If we change the replication factor or block size in configuration file, we have restart the cluster to apply the changes for coming Data.

Hadoop Distributed Filesystem

        When a dataset outgrows the storage capacity of a single physical machine, it becomes necessary to partition it across a number of separate machines. Filesystems that manage the storage across a network of machines are called distributed filesystems.

Apache Hadoop is an Open source Distributed system with fault tolerance Mechanism.

Hadoop comes with a distributed filesystem called HDFS, which stands for Hadoop Distributed Filesystem.

BasicTerminologies in Hadoop:

Cluster: All the nodes(systems) should be under common dedicated network. Minimum size of Cluster is “1”.

Hadoop Cluster: The Nodes should be appended with both HDFS and MAPREDUCE.

VPN: Virtual private Network.

The Design of HDFS

                            HDFS is a filesystem designed for storing very large files with streaming data access patterns, running on clusters of commodity hardware. Let’s examine this statement in more detail:

Very large files

              “Very large” in this context means files that are hundreds of megabytes, gigabytes, or terabytes in size. There are Hadoop clusters running today that store petabytes of data.

Streaming data access

 HDFS is built around the idea that the most efficient data processing pattern is a write-once, read-many-times pattern. A dataset is typically generated or copied from source, and then various analyses are performed on that dataset over time. Each analysis will involve a large proportion, if not all, of the dataset, so the time to read the whole dataset is more important than the latency in reading the first record.

Commodity hardware

                    Hadoop doesn’t require expensive, highly reliable hardware to run on. It’s designed to run on clusters of commodity hardware (commonly available hardware available from multiple vendors3) for which the chance of node failure across the cluster is high, at least for large clusters. HDFS is designed to carry on working without a noticeable interruption to the user in the face of such failure.

Low-latency data access

                   Applications that require low-latency access to data, in the tens of milliseconds range, will not work well with HDFS. Remember, HDFS is optimized for delivering a high throughput of data, and this may be at the expense of latency. HBase is currently a better choice for low-latency access.

Lots of small files

                    Since the namenode holds filesystem metadata in memory, the limit to the number of files in a filesystem is governed by the amount of memory on the namenode. As a rule of thumb, each file, directory, and block takes about 150 bytes. So, for example, if you had one million files, each taking one block, you would need at least 300 MB of memory. While storing millions of files is feasible, billions is beyond the capability of current hardware.

Write Once read many times

                             Files in HDFS may be written to by a single writer. Writes are always made at the end of the file. There is no support for multiple writers, or for modifications at arbitrary offsets in the file. (These might be supported in the future, but they are likely to be relatively inefficient.)

Moving Business logic near to the Data is easy than moving the Data to the Business logic

                      That means all the computations are moved to the Data rather than moving data near to the Business logic.

A Brief History of Hadoop

Hadoop was created by Doug Cutting, the creator of Apache Lucene, the widely used text search library. Hadoop has its origins in Apache Nutch, an open source web search engine, itself a part of the Lucene project.

          The project’s creator, Doug Cutting, explains how the name came about:

The name my kid gave a stuffed yellow elephant. Short, relatively easy to spell and pronounce, meaningless, and not used elsewhere: those are my naming criteria. Kids are good at generating such. Googol is a kid’s term.

            In January 2008, Hadoop was made its own top-level project at Apache, confirming its success and its diverse, active community. By this time, Hadoop was being used by many other companies besides Yahoo!, such as Last.fm, Facebook. In April 2008, Hadoop broke a world record to become the fastest system to sort a terabyte of data. Running on a 910-node cluster, Hadoop sorted one terabyte in 209 seconds (just under 3½ minutes).

Here’s a quick timeline of how thingshave progressed:

• 2004—Initial versions of what is now Hadoop Distributed Filesystem and Map-Reduce implemented by Doug Cutting and Mike Cafarella.

• December 2005—Nutch ported to the new framework. Hadoop runs reliably on 20 nodes.

• January 2006—Doug Cutting joins Yahoo!.

• February 2006—Apache Hadoop project officially started to support the standalone

development of MapReduce and HDFS.

• February 2006—Adoption of Hadoop by Yahoo! Grid team.

• April 2006—Sort benchmark (10 GB/node) run on 188 nodes in 47.9 hours.

• May 2006—Yahoo! set up a Hadoop research cluster—300 nodes.

• May 2006—Sort benchmark run on 500 nodes in 42 hours (better hardware than April benchmark).

• October 2006—Research cluster reaches 600 nodes.

• December 2006—Sort benchmark run on 20 nodes in 1.8 hours, 100 nodes in 3.3hours, 500 nodes in 5.2 hours, 900 nodes in 7.8 hours.

• January 2007—Research cluster reaches 900 nodes.

• April 2007—Research clusters—2 clusters of 1000 nodes.

• April 2008—Won the 1 terabyte sort benchmark in 209 seconds on 900 nodes.

• October 2008—Loading 10 terabytes of data per day on to research clusters.

• March 2009—17 clusters with a total of 24,000 nodes.

• April 2009—Won the minute sort by sorting 500 GB in 59 seconds (on 1,400 nodes) and the 100 terabyte sort in 173 minutes (on 3,400 nodes).

Why can’t we use databases with lots of disks to do large-scale batch analysis?

RDBMS

Why can’t we use databases with lots of disks to do large-scale batch analysis? Why is

Map Reduce needed?

Solution: RDBMS can’t scalable, Hadoop can easily scalable by adding no. of systems.

RDBMS can store only structured data, rather Hadoop can store any Data.(structured/Semi Structured/Un Structured).

	Traditional RDBMS	Map Reduce
Data size	Gigabytes	Petabytes
Access	Interactive and batch	Batch
Updates	Read and write many times	Write once, read many times
Structure	Static schema	Dynamic schema
Integrity	High	Low
Scaling	Nonlinear	Linear

Grid Computing

The High Performance Computing (HPC) and Grid Computing communities have been doing large-scale data processing for years, using such APIs as Message Passing Interface (MPI). Broadly, the approach in HPC is to distribute the work across a cluster

Of machines, which access a shared file system, hosted by a SAN. This works well for predominantly compute-intensive jobs, but becomes a problem when nodes need to access larger data volumes (hundreds of gigabytes, the point at which Map Reduce really

starts to shine), since the network bandwidth is the bottleneck and compute nodes become idle.

Volunteer Computing

                     Volunteer computing projects work by breaking the problem they are trying to solve into chunks called work units, which are sent to computers around the world to be analyzed. For example, a SETI@home work unit is about 0.35 MB of radio telescope data, and takes hours or days to analyze on a typical home computer. When the analysis

is completed, the results are sent back to the server, and the client gets another work unit. As a precaution to combat cheating, each work unit is sent to three different machines and needs at least two results to agree to be accepted. Although SETI@home may be superficially similar to Map Reduce (breaking a problem into independent pieces to be worked on in parallel), there are some significant differences. The SETI@home problem is very CPU-intensive, which makes it suitable for running on hundreds of thousands of computers across the world,since the time to transfer the work unit is dwarfed by the time to run the computation on it. Volunteers are donating CPU cycles, not bandwidth.

Data Storage and Analysis

Data!

We live in the data age. It’s not easy to measure the total volume of data stored electronically, but an IDC estimate put the size of the “digital universe” at 0.18 zettabytes in 2006, and is forecasting a tenfold growth by 2011 to 1.8 zettabytes.1 A zettabyte is 1021 bytes, or equivalently one thousand exabytes, one million petabytes, or one billion terabytes.

Example:  Take photos, for example. My wife’s grandfather was an avid photographer, and took photographs throughout his adult life. His entire corpus of medium format, slide, and 35mm film, when scanned in at high-resolution, occupies around 10 gigabytes.

Data Storage and Analysis

The problem is simple: while the storage capacities of hard drives have increased massively over the years, access speeds—the rate at which data can be read from drives—have not kept up. One typical drive from 1990 could store 1,370 MB of data and had a transfer speed of 4.4 MB/s, so you could read all the data from a full drive in around five minutes. Over 20 years later, one terabyte drives are the norm, but the transfer speed is around 100 MB/s, so it takes more than two and a half hours to read all the data off the disk.

Solution:

This is a long time to read all data on a single drive—and writing is even slower. The obvious way to reduce the time is to read from multiple disks at once. Imagine if we had 100 drives, each holding one hundredth of the data. Working in parallel, we could read the data in less than two minutes.

The first problem to solve is hardware failure: as soon as you start using many pieces

of hardware, the chance that one will fail is fairly high. A common way of avoiding data

loss is through replication: redundant copies of the data are kept by the system so that

in the event of failure, there is another copy available.

The second problem is that most analysis tasks need to be able to combine the data in

some way; data read from one disk may need to be combined with the data from any

of the other 99 disks. Various distributed systems allow data to be combined from

multiple sources, but doing this correctly is notoriously challenging.

This, in a nutshell, is what Hadoop provides: a reliable shared storage and analysis

system. Thestorage is provided by HDFS and analysis by Map Reduce.

What Do You Do with Data

Lets take an example:

                 Ø From application developers to video streamers, organizations of all sizes face the challenge of capturing, searching, analyzing, and leveraging as much as terabytes of data per second—too much for the constraints of traditional system capabilities and database management tools.”

Why Big Data ?

§  Key enablers for the appearance and growth of ‘Big-Data’ are:

+     Increase in storage capabilities

+     Increase in processing power

+     Availability of data

When dealing with Big Data is hard:   

§  When the operations on data are complex:

–     Eg. Simple counting is not a complex problem.

–     Modeling and reasoning with data of different kinds can get extremely complex

§  Good news with big-data:

–     Often, because of the vast amount of data, modeling techniques can get simpler (e.g., smart counting can replace complex model-based analytics)…

–     …as long as we deal with the scale.