Experimental Evaluation of Memory Configurations of Hadoop in Docker Environments

Hadoop is widely used in these days for big data analytics. Docker, a new container technology, is hot nowadays, and is the new QuickStart option for Apache Hadoop. It is a trend to build Hadoop cluster in Docker environments... [ view full abstract ]

Hadoop is widely used in these days for big data analytics. Docker, a new container technology, is hot nowadays, and is the new QuickStart option for Apache Hadoop. It is a trend to build Hadoop cluster in Docker environments in clouds or clusters. However, how to make better use of hardware resources and improve Hadoop performance in Docker environments is a challenge for users. In this paper we study memory configurations of Hadoop in Docker environments, and analyse the performance of Hadoop while altering Hadoop’s memory configurations. We select two different applications (a CPU-intensive application – WordCount and a memory-intensive application – TeraSort) and measure their resource usages in terms of CPU and memory footprint. Extensive tests have been executed, the test results show that appropriate customization of memory configurations improves performance compared to the default memory parameter settings.