A Brief History of Hadoop

Hadoop was created by Doug Cutting, the creator of Apache Lucene, the widely used text search library. Hadoop has its origins in Apache Nutch, an open source web search engine, itself a part of the Lucene project.

          The project’s creator, Doug Cutting, explains how the name came about:

The name my kid gave a stuffed yellow elephant. Short, relatively easy to spell and pronounce, meaningless, and not used elsewhere: those are my naming criteria. Kids are good at generating such. Googol is a kid’s term.

            In January 2008, Hadoop was made its own top-level project at Apache, confirming its success and its diverse, active community. By this time, Hadoop was being used by many other companies besides Yahoo!, such as Last.fm, Facebook. In April 2008, Hadoop broke a world record to become the fastest system to sort a terabyte of data. Running on a 910-node cluster, Hadoop sorted one terabyte in 209 seconds (just under 3½ minutes).

Here’s a quick timeline of how thingshave progressed:

• 2004—Initial versions of what is now Hadoop Distributed Filesystem and Map-Reduce implemented by Doug Cutting and Mike Cafarella.

• December 2005—Nutch ported to the new framework. Hadoop runs reliably on 20 nodes.

• January 2006—Doug Cutting joins Yahoo!.

• February 2006—Apache Hadoop project officially started to support the standalone

development of MapReduce and HDFS.

• February 2006—Adoption of Hadoop by Yahoo! Grid team.

• April 2006—Sort benchmark (10 GB/node) run on 188 nodes in 47.9 hours.

• May 2006—Yahoo! set up a Hadoop research cluster—300 nodes.

• May 2006—Sort benchmark run on 500 nodes in 42 hours (better hardware than April benchmark).

• October 2006—Research cluster reaches 600 nodes.

• December 2006—Sort benchmark run on 20 nodes in 1.8 hours, 100 nodes in 3.3hours, 500 nodes in 5.2 hours, 900 nodes in 7.8 hours.

• January 2007—Research cluster reaches 900 nodes.

• April 2007—Research clusters—2 clusters of 1000 nodes.

• April 2008—Won the 1 terabyte sort benchmark in 209 seconds on 900 nodes.

• October 2008—Loading 10 terabytes of data per day on to research clusters.

• March 2009—17 clusters with a total of 24,000 nodes.

• April 2009—Won the minute sort by sorting 500 GB in 59 seconds (on 1,400 nodes) and the 100 terabyte sort in 173 minutes (on 3,400 nodes).