Singularity container issue on `CLOUD.UKI-LT2-IC-HEP-lz.uk`
******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hi, I'm running a local singularity image in a dirac job, this job runs fine on `LCG.UKI-LT2-IC-HEP.uk` but when on `CLOUD.UKI-LT2-IC-HEP-lz.uk` I get a container creation failed: Error message: ``` INFO: Converting SIF file to temporary sandbox... FATAL: while extracting hyper_lite__0_1.sif: root filesystem extraction failed: extract command failed: FATAL: container creation failed: multiple devpts instances unsupported and /dev/pts configured : exit status 255 ``` This is my singularity command (mounting current working directory, x509 proxy location and cvmfs): ``` $SINGULARITY_BIN run --contain --env X509_USER_PROXY=${X509_USER_PROXY} \ -B $PWD -B $X509_USER_PROXY -B /cvmfs \ $LOCAL_IMAGE /bin/sh -l $JOB_SCRIPT ``` Also by default i'm using `/cvmfs/ oasis.opensciencegrid.org/mis/singularity/current/bin/singularity` for the singularity binary, unless user namespaces aren't enabled on that system and then it uses the local singularity instead. I wonder if the issue is with something i'm mounting or is it something else? Thanks, Andrew -- ____________________________ *Andrew Naylor* Astroparticle Physics PhD Student Department of Physics and Astronomy The University of Sheffield *Office*:E18a, Hicks Building, Hounsfield Road, Sheffield S3 7RH, UK *Email*:andrew.naylor@sheffield.ac.uk
Hi Andrew, On Fri, Aug 06, 2021 at 05:36:47PM +0100, Andrew S Naylor wrote:
FATAL: while extracting hyper_lite__0_1.sif: root filesystem extraction
The LZ cloud site uses singularity for payload isolation: It's still possible to run a "nested" singularity instance, but there are a few extra limitations.
failed: extract command failed: FATAL: container creation failed:
It looks like you're using a .sif image file: We generally suggest using an extracted image directory instead as it works more reliably (this is one of the limitations of nested singularity).
Also by default i'm using `/cvmfs/ oasis.opensciencegrid.org/mis/singularity/current/bin/singularity` for the singularity binary, unless user namespaces aren't enabled on that system and then it uses the local singularity instead.
Yes, that's a sensible approach. If you use the version from CVMFS, add the --userns option too, which (along with avoiding the .sif format) should fix the problem. We wrote a script encapsulating all of the singularity workarounds we know about for LZ: I've included that below for reference (there are also some quirks around the current working directory at some other sites). p.s. You're not actually subscribed to this mailing list, you may want to consider joining it; it's rather low traffic. This sign-up page is: https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users Regards, Simon #!/bin/bash # LZ Grid Container script # v1.0 ICHEP June 2021 - Initial version # v1.1 ICHEP June 2021 - Removed sysctl call and fixed /srv bug # Detect singularity location and run-mode SINGULARITY='/cvmfs/oasis.opensciencegrid.org/mis/singularity/current/bin/singularity' CONTAINER_OPT="" if [ -f /usr/bin/singularity ]; then # Use locally installed version SINGULARITY='/usr/bin/singularity' fi # Detect user namespace support and enable it if needed USERNS=$(awk '{ print $NF }' < /proc/sys/user/max_user_namespaces) if [ ${USERNS} -gt 200 ]; then CONTAINER_OPTS="--userns" fi # Uses the standard LZ container from CVMFS. CONTAINER='/cvmfs/singularity.opensciencegrid.org/luxzeplin/base_os:centos7' # Copy the proxy to the CWD if [ ! -z "${X509_USER_PROXY}" ]; then cp -p "${X509_USER_PROXY}" "${PWD}/user.proxy" elif [ -f "/tmp/x509up_u$(id -u)" ]; then # Proxy is using default location cp -p "/tmp/x509up_u$(id -u)" "${PWD}/user.proxy" fi # Prepare the environment for inside the conainter # Variables prepended with SINGULARITYENV_ are automatically included # even with the --cleanenv set. # Proxy inside the container needs to use the absolute /srv path export SINGULARITYENV_X509_USER_PROXY="/srv/user.proxy" export SINGULARITYENV_X509_CERT_DIR=/cvmfs/grid.cern.ch/etc/grid-security/certificates export SINGULARITYENV_LZ_SETUP_DBI=false # Mount the local path as /srv in the container and CVMFS CONTAINER_OPTS="${CONTAINER_OPTS} -B /cvmfs -B ${PWD}:/srv --pwd /srv --cleanenv" exec ${SINGULARITY} exec ${CONTAINER_OPTS} ${CONTAINER} "${@}"
participants (2)
-
Andrew S Naylor
-
Simon Fayer