Issues compiling on Lustre filesystem
Hi folks, I'm having issues compiling on York's new supercomputer (Viking - cool name). I have successfully compiled on the machine, but in the frontend /users space which is not available on the backend. When I compile on he backend filesystem, I get the error attached - basically a permission error when moving files The drive available on the backend is called scratch. I've tried a few things: - move the venv: virtualenv --relocatable firedrake, mv it, but this fails when running (just hangs somewhere in the simulation - probably mpipy) These all fail compilation: - set the TMPDIR flag to /user during compilation - compile on the back end - compile on the frontend directly on /scratch I suspect it's something to do with the Lustre filesystem as I can compile on the frontend space. Thanks in advance for any advice! Jon -- Dr Jon Hill Senior Lecturer in Physical Geography Department of Environment and Geography University of York M: +44(0)7748254812 P: +44(0)1904 324480 Twitter: @jonxhill Web: https://jonxhill.wordpress.com/ Web: https://envmodellinggroup.github.io/
Hi Jon, One option is to compile petsc separately and point firedrake at it. This is a common strategy many users adopt on slightly uncooperative systems. You can get the correct set of configure options from: firedrake-install --show-petsc-configure-options and you'll want to use the firedrake fork from https://github.com/firedrakeproject/petsc . I would normally think that compiling on the backend is the Right Thing. Can you post the error from that? (I think the error you posted was from a front end compilation). Cheers, David On 07/02/2019, 08:53, "firedrake-bounces@imperial.ac.uk on behalf of Jon Hill" <firedrake-bounces@imperial.ac.uk on behalf of jon.hill@york.ac.uk> wrote: Hi folks, I'm having issues compiling on York's new supercomputer (Viking - cool name). I have successfully compiled on the machine, but in the frontend /users space which is not available on the backend. When I compile on he backend filesystem, I get the error attached - basically a permission error when moving files The drive available on the backend is called scratch. I've tried a few things: - move the venv: virtualenv --relocatable firedrake, mv it, but this fails when running (just hangs somewhere in the simulation - probably mpipy) These all fail compilation: - set the TMPDIR flag to /user during compilation - compile on the back end - compile on the frontend directly on /scratch I suspect it's something to do with the Lustre filesystem as I can compile on the frontend space. Thanks in advance for any advice! Jon -- Dr Jon Hill Senior Lecturer in Physical Geography Department of Environment and Geography University of York M: +44(0)7748254812 P: +44(0)1904 324480 Twitter: @jonxhill Web: https://jonxhill.wordpress.com/ Web: https://envmodellinggroup.github.io/
Hi David, Thanks for the response. Backend error is the same, just /tmp/blah rather than my TMPDIR path. , e.g. shutil.Error: [('/mnt/lustre/users/jh1889/firedrake/src/petsc/.git/objects/pack/pack-e1302a45b9ef1bb694400dd8c57ee327e5f335b4.pack', '/tmp/pip-req-build-rflwokgy/.git/objects/pack/pack-e1302a45b9ef1bb694400dd8c57ee327e5f335b4.pack' Will try the petsc compilation separately. Ta, J. On 07/02/2019 09:50, Ham, David A wrote:
Hi Jon,
One option is to compile petsc separately and point firedrake at it. This is a common strategy many users adopt on slightly uncooperative systems. You can get the correct set of configure options from: firedrake-install --show-petsc-configure-options and you'll want to use the firedrake fork from https://github.com/firedrakeproject/petsc .
I would normally think that compiling on the backend is the Right Thing. Can you post the error from that? (I think the error you posted was from a front end compilation).
Cheers,
David
On 07/02/2019, 08:53, "firedrake-bounces@imperial.ac.uk on behalf of Jon Hill" <firedrake-bounces@imperial.ac.uk on behalf of jon.hill@york.ac.uk> wrote:
Hi folks,
I'm having issues compiling on York's new supercomputer (Viking - cool name). I have successfully compiled on the machine, but in the frontend /users space which is not available on the backend.
When I compile on he backend filesystem, I get the error attached - basically a permission error when moving files
The drive available on the backend is called scratch. I've tried a few things: - move the venv: virtualenv --relocatable firedrake, mv it, but this fails when running (just hangs somewhere in the simulation - probably mpipy)
These all fail compilation: - set the TMPDIR flag to /user during compilation - compile on the back end - compile on the frontend directly on /scratch
I suspect it's something to do with the Lustre filesystem as I can compile on the frontend space.
Thanks in advance for any advice!
Jon
-- Dr Jon Hill Senior Lecturer in Physical Geography Department of Environment and Geography University of York M: +44(0)7748254812 P: +44(0)1904 324480 Twitter: @jonxhill Web: https://jonxhill.wordpress.com/ Web: https://envmodellinggroup.github.io/
-- Dr Jon Hill Senior Lecturer in Physical Geography Chair of Dept. Athena Swan Committee Department of Environment and Geography University of York M: +44(0)7748254812 P: +44(0)1904 324480 Twitter: @jonxhill Web: https://jonxhill.wordpress.com/ Web: https://envmodellinggroup.github.io/
hi folks, made some progress on this. Built petsc manually, still fails with same permission error. Found this bug report for pip install on panfs: https://github.com/pypa/pip/issues/2941 Turns out setting write flags on the git pack idx and .pack files works on lustre too. I could then (within the activated firedrake venv) do the pip install. Without the write flags, I got the same error when running pip install manually within the venv. I'll hack the firedrake-install script to do the chmod once petsc4py has been cloned and hopefully this will work for a full install. Will let you know if that works. Cheers, J. On 07/02/2019 09:50, Ham, David A wrote:
Hi Jon,
One option is to compile petsc separately and point firedrake at it. This is a common strategy many users adopt on slightly uncooperative systems. You can get the correct set of configure options from: firedrake-install --show-petsc-configure-options and you'll want to use the firedrake fork from https://github.com/firedrakeproject/petsc .
I would normally think that compiling on the backend is the Right Thing. Can you post the error from that? (I think the error you posted was from a front end compilation).
Cheers,
David
On 07/02/2019, 08:53, "firedrake-bounces@imperial.ac.uk on behalf of Jon Hill" <firedrake-bounces@imperial.ac.uk on behalf of jon.hill@york.ac.uk> wrote:
Hi folks,
I'm having issues compiling on York's new supercomputer (Viking - cool name). I have successfully compiled on the machine, but in the frontend /users space which is not available on the backend.
When I compile on he backend filesystem, I get the error attached - basically a permission error when moving files
The drive available on the backend is called scratch. I've tried a few things: - move the venv: virtualenv --relocatable firedrake, mv it, but this fails when running (just hangs somewhere in the simulation - probably mpipy)
These all fail compilation: - set the TMPDIR flag to /user during compilation - compile on the back end - compile on the frontend directly on /scratch
I suspect it's something to do with the Lustre filesystem as I can compile on the frontend space.
Thanks in advance for any advice!
Jon
-- Dr Jon Hill Senior Lecturer in Physical Geography Department of Environment and Geography University of York M: +44(0)7748254812 P: +44(0)1904 324480 Twitter: @jonxhill Web: https://jonxhill.wordpress.com/ Web: https://envmodellinggroup.github.io/
-- Dr Jon Hill Senior Lecturer in Physical Geography Department of Environment and Geography University of York M: +44(0)7748254812 P: +44(0)1904 324480 Twitter: @jonxhill Web: https://jonxhill.wordpress.com/ Web: https://envmodellinggroup.github.io/
Hi folks, Just to confirm this is indeed the case. I could compile using: python3 firedrake-install --install thetis --no-package-manager --disable-ssh and run the barotropicChannel test in thetis in parallel (though I note it failed: volume3d rel. error 5.1939e-12 assert vol3d_rerr < 1e-12 ) I added a dirty hack to the firedrake-install script: def install(package): log.info("Installing %s" % package) if package == "petsc/": # hack to fix permission errors on Lustre file systems for dirpath, dirnames, filenames in os.walk( os.path.join(firedrake_env, "src",package,".git/objects/pack/")): for filename in filenames: path = os.path.join(dirpath, filename) print(path) os.chmod(path, 0o664) build_and_install_petsc() # The following outrageous hack works around the fact that petsc and co. cannot be installed in developer mode. elif package not in ["petsc4py/", "slepc/", "slepc4py/"]: run_pip_install(["-e", package]) else: if package == "petsc4py/" or package == "petsc/": # hack to fix permission errors on Lustre file systems for dirpath, dirnames, filenames in os.walk( os.path.join(firedrake_env, "src",package,".git/objects/pack/")): for filename in filenames: path = os.path.join(dirpath, filename) print(path) os.chmod(path, 0o644) run_pip_install(["--ignore-installed", package]) I imagine this could be tidied up a lot and only ran if the filesystem is Lustre or panfs. Just emailing the list so this is recorded if anyone else has this issue. Cheer and have a nice weekend, Jon On 08/02/2019 08:55, Jon Hill wrote:
hi folks,
made some progress on this. Built petsc manually, still fails with same permission error.
Found this bug report for pip install on panfs: https://github.com/pypa/pip/issues/2941
Turns out setting write flags on the git pack idx and .pack files works on lustre too. I could then (within the activated firedrake venv) do the pip install. Without the write flags, I got the same error when running pip install manually within the venv.
I'll hack the firedrake-install script to do the chmod once petsc4py has been cloned and hopefully this will work for a full install. Will let you know if that works.
Cheers, J.
On 07/02/2019 09:50, Ham, David A wrote:
Hi Jon,
One option is to compile petsc separately and point firedrake at it. This is a common strategy many users adopt on slightly uncooperative systems. You can get the correct set of configure options from: firedrake-install --show-petsc-configure-options and you'll want to use the firedrake fork from https://github.com/firedrakeproject/petsc .
I would normally think that compiling on the backend is the Right Thing. Can you post the error from that? (I think the error you posted was from a front end compilation).
Cheers,
David
On 07/02/2019, 08:53, "firedrake-bounces@imperial.ac.uk on behalf of Jon Hill" <firedrake-bounces@imperial.ac.uk on behalf of jon.hill@york.ac.uk> wrote:
Hi folks, I'm having issues compiling on York's new supercomputer (Viking - cool name). I have successfully compiled on the machine, but in the frontend /users space which is not available on the backend. When I compile on he backend filesystem, I get the error attached - basically a permission error when moving files The drive available on the backend is called scratch. I've tried a few things: - move the venv: virtualenv --relocatable firedrake, mv it, but this fails when running (just hangs somewhere in the simulation - probably mpipy) These all fail compilation: - set the TMPDIR flag to /user during compilation - compile on the back end - compile on the frontend directly on /scratch I suspect it's something to do with the Lustre filesystem as I can compile on the frontend space. Thanks in advance for any advice! Jon -- Dr Jon Hill Senior Lecturer in Physical Geography Department of Environment and Geography University of York M: +44(0)7748254812 P: +44(0)1904 324480 Twitter: @jonxhill Web: https://jonxhill.wordpress.com/ Web: https://envmodellinggroup.github.io/
-- Dr Jon Hill Senior Lecturer in Physical Geography Chair of Dept. Athena Swan Committee Department of Environment and Geography University of York M: +44(0)7748254812 P: +44(0)1904 324480 Twitter: @jonxhill Web: https://jonxhill.wordpress.com/ Web: https://envmodellinggroup.github.io/
participants (2)
-
Ham, David A
-
Jon Hill