Using the Raspberry Pi and Docker for Replicable Performance Experiments: Experience Paper

Knoche, Holger and Eichelberger, Holger (2018) Using the Raspberry Pi and Docker for Replicable Performance Experiments: Experience Paper [Paper] In: ACM/SPEC International Conference on Performance Engineering Proceedings of the 2018, April 09 - 13, 2018, Berlin.

ICPE2018-Knoche.pdf - Published Version

Download (1067Kb) | Preview
ICPE2018-Knoche-Presentation.pdf - Presentation

Download (724Kb) | Preview

Supplementary data:


Replicating software performance experiments is difficult. A common obstacle to replication is that recreating the hardware and software environments is often impractical. As researchers usually run their experiments on the hardware and software that happens to be available to them, recreating the experiments would require obtaining identical hardware, which can lead to high costs. Recreating the software environment is also difficult, as software components such as particular library versions might no longer be available. Cheap, standardized hardware components like the Raspberry Pi and portable software containers like the ones provided by Docker are a potential solution to meet the challenge of replicability. In this paper, we report on experiences from replicating performance experiments on Raspberry Pi devices with and without Docker and show that good replication results can be achieved for microbenchmarks such as JMH. Replication of macrobenchmarks like SPECjEnterprise 2010 proves to be much more difficult, as they are strongly affected by (non-standardized) peripherals. Inspired by previous microbenchmarking experiments on the Pi platform, we furthermore report on a systematic analysis of response time fluctuations, and present lessons learned on dos and don'ts for replicable performance experiments.

Document Type: Conference or Workshop Item (Paper)
Keywords: Raspberry Pi, performance benchmark, replicability, single-board computer
Research affiliation: Kiel University > Software Engineering
DOI etc.: 10.1145/3184407.3184431
Projects: Kieker
Date Deposited: 09 Apr 2018 11:31
Last Modified: 04 May 2018 10:37

Actions (login required)

View Item View Item

Document Downloads

More statistics for this item...