Sanjay Sharma’s Weblog

November 18, 2010

Multicore impact on software development

Filed under: Uncategorized — indoos @ 4:52 pm

Recently got the chance to rub shoulders with academia and chip-design software engineers in Bengaluru at http://www.innovate-it.in/ – a conference on “Issues in design of complex Multi-Core Systems”. I was speaking on “Multicore:Choice of Middleware and Framework” and was one of the few ones from the application software world whilst most were experts from the chip design or hard-core hardware/system level programming background.

Few of my revelations from the event were

– There are no silver bullets (yet!) for migrating traditional software to multi-core

– There is certainly a huge vacant play ground for new Players to come up with technologies to allow software to harness muti-core power with no/minimal software changes. Azul Systems is a real world example of this.

– One interesting finding was that HADOOP design is very much similar to multi-core internals. Distribution of work, data sharing/sync and better cache management are problems common to both and being solved almost in the same fashion. Nice to know that basic fundamental solutions fit in at Mega levels as well as Micro levels!

Hadoop’s programming model fits quite well in multi-core world as evident by some success reported on running Hadoop on GPU (MARS).

One of the practical tips for Hadoop clusters is to keep the count of max maps+reducers  on single node depending on the number of cores. 8 core machines can  run more parallel maps+reducers  than 4 core machines.  Also keep in  mind that data node and task tracker would also be consuming some of the cores’ resources.  Refer to this paper for more details-  White paper-HadoopPerformanceTuning.pdf

Advertisements

1 Comment »

  1. Another interesting side effect of multicore processor design is the rise of functional languages likes Haskell and F# which, thanks to their pure functions and immutable data, are intrinsically suited for parallel processing. Hadoop’s MR programming model borrows heavily from the map and reduce functions found in almost all of functional languages.

    Comment by Anand Nalya — July 28, 2011 @ 4:59 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: