White Papers

ManualsBrandsDell ManualsConverged InfrastructureHigh Performance Computing Solution Resources

Ready Solutions Engineering Test Results

Containerizing HPC Applications with Singularity

Authors: Rengan Xu, Frank Han, Nishanth Dandapanthula.

HPC Innovation Lab. October 2017

Overview

In this blog, we will give an introduction to Singularity containers and how they should be used to containerize HPC applications. We run

different deep learning frameworks with and without Singularity containers and show that there is no performance loss with Singularity

containers. We also show that Singularity can be easily used to run MPI applications.

Introduction to Singularity

Singularity is a container system developed by Lawrence Berkeley Lab to provide container technology like Docker for High Performance

Computing (HPC). It wraps applications into an isolated virtual environment to simplify application deployment. Unlike virtual machines,

the container does not have a virtual hardware layer and its own Linux kernel inside the host OS. It is just sandboxing the environment;

therefore, the overhead and the performance loss are minimal. The goal of the container is reproducibility. The container has all

environment and libraries an application needs to run, and it can be deployed anywhere so that anyone can reproduce the results the

container creator generated for that application.

Besides Singularity, another popular container is Docker, which has been widely used for many applications. However, there are several

reasons that Docker is not suitable for an HPC environment. The following are various reasons that we choose Singularity rather than

Docker:

 Security concern. Because of Docker daemon, a user inside the Docker container is able to obtain root access on the host and

then may act maliciously on the supercomputing cluster. In contrast, Singularity solves this by running the container with the

user’s credentials. The access permissions of a user are the same both inside the container and outside the container. Thus, a

non-root user cannot change anything outside of his/her permission.

 HPC Scheduler. Docker does not support any HPC job scheduler, but Singularity integrates seamlessly with all job schedulers

including SLURM, Torque, SGE, etc.

 GPU support. Docker does not support GPU natively. Nvidia Docker is a GPU-enabled Docker container, but it preinstalls

various software that a user may not need. Singularity is able to support GPUs natively. Users can install whatever CUDA version

and software they want on the host which can be transparently passed to Singularity.

MPI support. Docker does not support MPI natively. So if a user wants to use MPI with Docker, a MPI-enabled Docker needs

to be developed. If a MPI-enabled Docker is available, the network stacks such as TCP and those needed by MPI are private to

the container which makes Docker containers not suitable for more complicated networks like Infiniband. In Singularity, the

user’s environment is shared to the container seamlessly.

Summary of content (4 pages)

PAGE 1
Ready Solutions Engineering Test Results 1 Containerizing HPC Applications with Singularity Authors: Rengan Xu, Frank Han, Nishanth Dandapanthula. HPC Innovation Lab. October 2017 Overview In this blog, we will give an introduction to Singularity containers and how they should be used to containerize HPC applications. We run different deep learning frameworks with and without Singularity containers and show that there is no performance loss with Singularity containers.
PAGE 2
Ready Solutions Engineering Test Results 2 Challenges with Singularity in HPC and Workaround Many HPC applications, especially deep learning applications, have deep library dependences and it is time consuming to figure out these dependences and debug build issues. Most deep learning frameworks are developed in Ubuntu but they need to be deployed to Red Hat Enterprise Linux. So it is beneficial to build those applications once in a container and then deploy them anywhere.
PAGE 3
Ready Solutions Engineering Test Results 2 Inception-V3 with ILSVRC2012 Dataset Singularity over bare-metal 1.20 1.10 1.00 1.00 1.01 1.01 1.00 1.00 1 P100 2 P100 1.00 1.00 0.96 0.98 0.90 0.80 0.70 0.60 0.50 NV-Caffe2 MXNet 4 P100 TensorFlow Figure 1: Performance comparison with and without Singularity Singularity at Scale We ran HPL across multiple nodes and compared the performance with and without container.
PAGE 4
Ready Solutions Engineering Test Results 2 In this blog, we introduced what Singularity is and how it can be used to containerize HPC applications. We discussed the benefits of using Singularity over Docker. We also mentioned the challenges of using Singularity in a cluster environment and the workarounds available. We compared the performance of bare metal vs Singularity container and the results indicated that there is no performance loss when using Singularity.