Performance Tuning of Hadoop MapReduce: A Noisy Gradient Approach
  Hadoop MapReduce is a framework for distributed storage and processing of
large datasets that is quite popular in big data analytics. It has various
configuration parameters (knobs) which play an important role in deciding the
performance i.e., the execution time of a given big data processing job.
Default values of these parameters do not always result in good performance and
hence it is important to tune them. However, there is inherent difficulty in
tuning the parameters due to two important reasons - firstly, the parameter
search space is large and secondly, there are cross-parameter interactions.
Hence, there is a need for a dimensionality-free method which can automatically
tune the configuration parameters by taking into account the cross-parameter
dependencies. In this paper, we propose a novel Hadoop parameter tuning
methodology, based on a noisy gradient algorithm known as the simultaneous
perturbation stochastic approximation (SPSA). The SPSA algorithm tunes the
parameters by directly observing the performance of the Hadoop MapReduce
system. The approach followed is independent of parameter dimensions and
requires only $2$ observations per iteration while tuning. We demonstrate the
effectiveness of our methodology in achieving good performance on popular
Hadoop benchmarks namely \emph{Grep}, \emph{Bigram}, \emph{Inverted Index},
\emph{Word Co-occurrence} and \emph{Terasort}. Our method, when tested on a 25
node Hadoop cluster shows 66\% decrease in execution time of Hadoop jobs on an
average, when compared to the default configuration. Further, we also observe a
reduction of 45\% in execution times, when compared to prior methods.
