Sanjay Sharma’s Weblog

August 16, 2010

Hadoop Ecosystem World-Map

While preparing for the keynote for the  recently held HUG India meetup on 31st July, I decided that I will try to keep my session short, but useful and relevant to the lined up sesssions on hiho, JAQL and Visual hive. I have always been a keen student of geography (still take pride in it!) and thought it would be great to draw a visual geographical map of Hadoop ecosystem. Here is what I came up with a little nice story behind it-

  1. How did it all start- huge data on the web!
  2. Nutch built to crawl this web data
  3. Huge data had to saved- HDFS was born!
  4. How to use this data?
  5. Map reduce framework built for coding and running analytics – java, any language-streaming/pipes
  6. How to get in unstructured data – Web logs, Click streams, Apache logs, Server logs  – fuse,webdav, chukwa, flume, Scribe
  7. Hiho and sqoop for loading data into HDFS – RDBMS can join the Hadoop band wagon!
  8. High level interfaces required over low level map reduce programming– Pig, Hive, Jaql
  9. BI tools with advanced UI reporting- drilldown etc- Intellicus 
  10. Workflow tools over Map-Reduce processes and High level languages
  11. Monitor and manage hadoop, run jobs/hive, view HDFS – high level view- Hue, karmasphere, eclipse plugin, cacti, ganglia
  12. Support frameworks- Avro (Serialization), Zookeeper (Coordination)
  13. More High level interfaces/uses- Mahout, Elastic map Reduce
  14. OLTP- also possible – Hbase

Would love to hear feedback about this and how to grow it further to add the missing parts!

Hadoop ecosystem map


  1. great work! can you add the various flavors and/or commercial implementations of these tools? e.g. greenplum, hypertable, cassandra, etc.?

    Comment by v — August 18, 2010 @ 5:13 am

  2. Hello Sanjay. The eco-system has come out very well and is very comprehensive. It’d have been great if I had attended the event. Good work. Keep it up.

    Comment by balamurugan — August 18, 2010 @ 5:14 am

  3. […] Hadoop Ecosystem World-Map « Sanjay Sharma’s Weblog (tags: hadoop) […]

    Pingback by links for 2010-08-18 | — August 19, 2010 @ 12:06 am

  4. […] Hadoop ecosystem World Map – e.g., hiho and Sqoop for loading RDBMS data into Hadoop. […]

    Pingback by State of Data Last Week – Aug 22 « Dr Data's Blog — August 22, 2010 @ 11:46 pm

  5. […] Hadoop Ecosystem World-Map « Sanjay Sharma’s Weblog […]

    Pingback by Hadoop: Links, News and Resources (1) « Angel “Java” Lopez on Blog — August 15, 2011 @ 9:55 am

  6. Great work! Can you add the elaboration of the each module.

    Comment by Surendra — December 14, 2011 @ 8:47 am

  7. I would have grouped Zookeeper and upcoming Apache Ambari in the coordination framework. Avro/Thrift and Protobuf as the serialization.

    Comment by Vikas Deolaliker — February 20, 2012 @ 10:10 pm

  8. Hello Sanjay, I plan to include link to your blog post about Hadoop ecosystem at my blog – I will be giving your reference as well. I hope that is OK. Please let me know otherwise.

    Comment by spawgi — September 23, 2012 @ 9:31 am

    • It would be a pleasure!

      Comment by indoos — September 23, 2012 @ 4:58 pm

      • I would also be referencing your ecosystem map to one of our presentation to our group for an intro to hadoop . is that okay?

        Comment by Elizabeth — April 11, 2013 @ 5:23 pm

  9. This is wonderful. Missing a few things like Tez, and also perhaps Lucene/Solr (not technically part of Apache Hadoop, but integral, IMHO). May I use this and build on it?

    Comment by Sue Feldman — July 10, 2014 @ 11:22 pm

  10. Hi Sanjay,
    I’d like to include this map in Entelo’s webinar on recruiting data scientists. We will include a link/reference to your name and this post. Do we have your permission to use your content? Please email me at

    Thank you!

    Comment by kathleen d — November 18, 2014 @ 12:57 am

    • Welcome! Please go ahead.

      Comment by indoos — November 18, 2014 @ 2:46 am

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The WordPress Classic Theme. Create a free website or blog at


Get every new post delivered to your Inbox.

Join 43 other followers

%d bloggers like this: