Sanjay Sharma’s Weblog

August 16, 2010

Hadoop Ecosystem World-Map

While preparing for the keynote for the  recently held HUG India meetup on 31st July, I decided that I will try to keep my session short, but useful and relevant to the lined up sesssions on hiho, JAQL and Visual hive. I have always been a keen student of geography (still take pride in it!) and thought it would be great to draw a visual geographical map of Hadoop ecosystem. Here is what I came up with a little nice story behind it-

  1. How did it all start- huge data on the web!
  2. Nutch built to crawl this web data
  3. Huge data had to saved- HDFS was born!
  4. How to use this data?
  5. Map reduce framework built for coding and running analytics – java, any language-streaming/pipes
  6. How to get in unstructured data – Web logs, Click streams, Apache logs, Server logs  – fuse,webdav, chukwa, flume, Scribe
  7. Hiho and sqoop for loading data into HDFS – RDBMS can join the Hadoop band wagon!
  8. High level interfaces required over low level map reduce programming– Pig, Hive, Jaql
  9. BI tools with advanced UI reporting- drilldown etc- Intellicus 
  10. Workflow tools over Map-Reduce processes and High level languages
  11. Monitor and manage hadoop, run jobs/hive, view HDFS – high level view- Hue, karmasphere, eclipse plugin, cacti, ganglia
  12. Support frameworks- Avro (Serialization), Zookeeper (Coordination)
  13. More High level interfaces/uses- Mahout, Elastic map Reduce
  14. OLTP- also possible – Hbase

Would love to hear feedback about this and how to grow it further to add the missing parts!

Hadoop ecosystem map

About these ads

14 Comments »

  1. great work! can you add the various flavors and/or commercial implementations of these tools? e.g. greenplum, hypertable, cassandra, etc.?

    Comment by v — August 18, 2010 @ 5:13 am

  2. Hello Sanjay. The eco-system has come out very well and is very comprehensive. It’d have been great if I had attended the event. Good work. Keep it up.

    Comment by balamurugan — August 18, 2010 @ 5:14 am

  3. […] Hadoop Ecosystem World-Map « Sanjay Sharma’s Weblog (tags: hadoop) […]

    Pingback by links for 2010-08-18 | andy.edmonds.be — August 19, 2010 @ 12:06 am

  4. […] Hadoop ecosystem World Map – e.g., hiho and Sqoop for loading RDBMS data into Hadoop. […]

    Pingback by State of Data Last Week – Aug 22 « Dr Data's Blog — August 22, 2010 @ 11:46 pm

  5. […] Hadoop Ecosystem World-Map « Sanjay Sharma’s Weblog http://indoos.wordpress.com/2010/08/16/hadoop-ecosystem-world-map/ […]

    Pingback by Hadoop: Links, News and Resources (1) « Angel “Java” Lopez on Blog — August 15, 2011 @ 9:55 am

  6. Great work! Can you add the elaboration of the each module.

    Comment by Surendra — December 14, 2011 @ 8:47 am

  7. I would have grouped Zookeeper and upcoming Apache Ambari in the coordination framework. Avro/Thrift and Protobuf as the serialization.

    Comment by Vikas Deolaliker — February 20, 2012 @ 10:10 pm

  8. Hello Sanjay, I plan to include link to your blog post about Hadoop ecosystem at my blog – http://spawgi.wordpress.com. I will be giving your reference as well. I hope that is OK. Please let me know otherwise.

    Comment by spawgi — September 23, 2012 @ 9:31 am

    • It would be a pleasure!

      Comment by indoos — September 23, 2012 @ 4:58 pm

      • I would also be referencing your ecosystem map to one of our presentation to our group for an intro to hadoop . is that okay?

        Comment by Elizabeth — April 11, 2013 @ 5:23 pm

  9. This is wonderful. Missing a few things like Tez, and also perhaps Lucene/Solr (not technically part of Apache Hadoop, but integral, IMHO). May I use this and build on it?

    Comment by Sue Feldman — July 10, 2014 @ 11:22 pm

  10. Hi Sanjay,
    I’d like to include this map in Entelo’s webinar on recruiting data scientists. We will include a link/reference to your name and this post. Do we have your permission to use your content? Please email me at kathleen@entelo.com

    Thank you!

    Comment by kathleen d — November 18, 2014 @ 12:57 am

    • Welcome! Please go ahead.

      Comment by indoos — November 18, 2014 @ 2:46 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The WordPress Classic Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 42 other followers

%d bloggers like this: