Provide a Singularity Images Cache

When the cluster does not have Internet access or if you want to provide a local cache of Singularity images to be used on the cluster, you can put the Singularity images you want in the applicable directory under the singularity_image_root directory. If found, the image reference is translated to a local file path and substituted in the singularity run command to avoid the need for Singularity to download and convert the image for each user.

Each version of Determined utilizes specifically-tagged Docker containers. The image tags referenced by default in this version of Determined are described below.

Default Docker Images

Environment

File Name

CPUs

determinedai/environments:py-3.8-pytorch-1.10-tf-2.8-cpu-096d730

Nvidia GPUs

determinedai/environments:cuda-11.3-pytorch-1.10-tf-2.8-gpu-096d730

AMD GPUs

determinedai/environments:rocm-5.0-pytorch-1.10-tf-2.7-rocm-096d730

See Set Environment Images for the images Docker Hub location and add each tagged image required by your environment and experiments to the image cache.

Add Images

Add each tagged image requried by your environment and the needs of your experiments to the image cache:

  1. Create a directory path using the same prefix as the image name referenced in the singularity_image_root directory. For example, the image determinedai/environments:cuda-11.3-pytorch-1.10-tf-2.8-gpu-9119094 is added in the directory determinedai.

    cd $singularity_image_root
    mkdir determinedai
    
  2. If your system has internet access, you can download images directly into the cache.

    cd $singularity_image_root
    image="determinedai/environments:cuda-11.3-pytorch-1.10-tf-2.8-gpu-9119094"
    singularity pull $image docker://$image
    
  3. Otherwise, from an internet-connected system, download the desired image using the Singulartity pull command then copy it to the determinedai folder under singularity_image_root.

    singularity pull \
          temporary-image \
          docker://determinedai/environments:cuda-11.3-pytorch-1.10-tf-2.8-gpu-9119094
    scp temporary-image mycluster:$singularity_image_root/determinedai/environments:cuda-11.3-pytorch-1.10-tf-2.8-gpu-9119094