How to Set Number of Hadoop Reducers on Command Line

Setting the number of reducers for a Hadoop MapReduce job can be very important. Fortunately, there is an easy way to to this from the command line using the -D <property=value> option.

Using -D Option on the Command Line

A simple example of the -D option to set the number of reducers to 10:
-D mapreduce.job.reduces=10

Word Count Example Default Reducers

An “examples” jar comes available in most Hadoop installations. This jar comes with various examples of using MapReduce for different tasks. Different versions of these jars, like this one, can be found in the Maven Repository. Here we see an example of running the the wordcount example without setting the number of reducers:

hadoop jar hadoop-mapreduce-examples.jar wordcount /myInputDir /myOutputDir


Word Count Example after Setting Reducers

Next we will run the same example setting the number of reducers to 10:

hadoop jar /usr/jars/hadoop-mapreduce-examples.jar wordcount -D mapreduce.job.reduces=10 /myInputDir /myOutputDir


Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">