Re: [firedrake] Installing firedrake on a HPC system without package manager
Hi Niall, (adding in firedrake cc again)
On 1 Aug 2019, at 15:43, Madden, Niall <niall.madden@nuigalway.ie> wrote:
Hi Lawrence, Thanks for the swift reply. I don't want to reply to the whole list with too much specific information on the system
Please do reply with the list. That way others who might be able to help will see things.
Can you give us some details on this system? Is it a cray, for example? The system is ICHEC's "kay" - Ireland's national supercomputer for academic purposes: https://www.ichec.ie/about/infrastructure/kay
Normally we find when building on supercomputers (contrary to the one-shot "firedrake-install" unfortunately), that it is best to build PETSc by hand. One then sets the appropriate environment variables and runs firedrake-install --honour-petsc-dir. This might be the best step forward.
Thanks for the tip. I tried compiling PETSc myself:
git clone https://github.com/firedrakeproject/petsc.git (but should I be cloning fromhttps://bitbucket.org/petsc/petsc ?)
No, cloning from the firedrakeproject "fork" is correct. We keep a (reasonably up to date) version of petsc that is known good with all Firedrake components there: it isn't otherwise special.
./configure --with-c2html=0 --download-chaco --download-metis --download-parmetis --download-scalapack --download-hypre --download-mumps --download-netcdf --download-hdf5 --download-pnetcdf --download-exodusii --download-mpich --download-suitesparse=1 --download-petsc4py=1 --download-mpi4py=1 --download-fblaslapack --download-zlib
Then I set PETSC_DIR and PETSC_ARCH, and PYTHONPATH
Then I try to install Firedrake with python3 firedrake-install --no-package-manager -v --disable-ssh --honour-petsc-dir --honour-pythonpath
But it eventually fails compiling libspatialindex for reasons that seem to be related to the mpicxx , but it is a little opaque: undefined reference to `psm2_info_query@PSM2_1.0'
Can you attach the contents of firedrake-install.log (maybe gzip it since it is likely to be large). Thanks, Lawrence
Hi Lawrence, Thanks for the pointer on PETSc. I was able to make and test it without problems. Then I set PETSC_DIR and PETSC_ARCH, and ran python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir It fails at subprocess.CalledProcessError: Command '['/MYOB/r/libs/firedrake/bin/python', '-m', 'pip', 'install', '--no-binary', 'mpi4py,randomgen', '-vvv', '-r', 'PyOP2/requirements-ext.txt']' returned non-zero exit status 1. But /MYOB/r/libs/firedrake/bin/python -m pip install --no-binary mpi4py -vvv -r ./src/PyOP2/requirements-ext.txt returns numerous "Requirement already satisfied:" messages. Gzip'ed log file attached. Thanks, Niall. ________________________________ From: Lawrence Mitchell <wence@gmx.li> Sent: Thursday 1 August 2019 15:53 To: Madden, Niall <niall.madden@nuigalway.ie> Cc: Firedrake Project <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, (adding in firedrake cc again)
On 1 Aug 2019, at 15:43, Madden, Niall <niall.madden@nuigalway.ie> wrote:
Hi Lawrence, Thanks for the swift reply. I don't want to reply to the whole list with too much specific information on the system
Please do reply with the list. That way others who might be able to help will see things.
Can you give us some details on this system? Is it a cray, for example? The system is ICHEC's "kay" - Ireland's national supercomputer for academic purposes: https://www.ichec.ie/about/infrastructure/kay
Normally we find when building on supercomputers (contrary to the one-shot "firedrake-install" unfortunately), that it is best to build PETSc by hand. One then sets the appropriate environment variables and runs firedrake-install --honour-petsc-dir. This might be the best step forward.
Thanks for the tip. I tried compiling PETSc myself:
git clone https://github.com/firedrakeproject/petsc.git (but should I be cloning fromhttps://bitbucket.org/petsc/petsc ?)
No, cloning from the firedrakeproject "fork" is correct. We keep a (reasonably up to date) version of petsc that is known good with all Firedrake components there: it isn't otherwise special.
./configure --with-c2html=0 --download-chaco --download-metis --download-parmetis --download-scalapack --download-hypre --download-mumps --download-netcdf --download-hdf5 --download-pnetcdf --download-exodusii --download-mpich --download-suitesparse=1 --download-petsc4py=1 --download-mpi4py=1 --download-fblaslapack --download-zlib
Then I set PETSC_DIR and PETSC_ARCH, and PYTHONPATH
Then I try to install Firedrake with python3 firedrake-install --no-package-manager -v --disable-ssh --honour-petsc-dir --honour-pythonpath
But it eventually fails compiling libspatialindex for reasons that seem to be related to the mpicxx , but it is a little opaque: undefined reference to `psm2_info_query@PSM2_1.0'
Can you attach the contents of firedrake-install.log (maybe gzip it since it is likely to be large). Thanks, Lawrence
Dear Niall, It seems to me the following line is critical: src/dynload.c:5:10: fatal error: Python.h: No such file or directory You want to have an appropriate package loaded. If you have the `module` command available, you could try `module search python3-dev` or (`module search python-dev`), followed by `module load package_name`. We can also see that you have openmpi loaded (`module list` will show you openmpi), but it is known to cause some issues, so you probably want to unload openmpi and load mpich. Though it is not directly related to the error you are having right now, petsc usually requires some additional parameters when configuring on clusters (--with-batch --known-mpi-shared-libraries=0) (https://www.mcs.anl.gov/petsc/documentation/installation.html ), <https://www.mcs.anl.gov/petsc/documentation/installation.html> which we don't see in your previous email. Were you able to test petsc on a compute node? If it worked on a computed node, I think it is fine. Regards, Koki ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Madden, Niall <niall.madden@nuigalway.ie> Sent: Thursday, August 1, 2019 8:56:26 PM To: Lawrence Mitchell <wence@gmx.li> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Lawrence, Thanks for the pointer on PETSc. I was able to make and test it without problems. Then I set PETSC_DIR and PETSC_ARCH, and ran python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir It fails at subprocess.CalledProcessError: Command '['/MYOB/r/libs/firedrake/bin/python', '-m', 'pip', 'install', '--no-binary', 'mpi4py,randomgen', '-vvv', '-r', 'PyOP2/requirements-ext.txt']' returned non-zero exit status 1. But /MYOB/r/libs/firedrake/bin/python -m pip install --no-binary mpi4py -vvv -r ./src/PyOP2/requirements-ext.txt returns numerous "Requirement already satisfied:" messages. Gzip'ed log file attached. Thanks, Niall. ________________________________ From: Lawrence Mitchell <wence@gmx.li> Sent: Thursday 1 August 2019 15:53 To: Madden, Niall <niall.madden@nuigalway.ie> Cc: Firedrake Project <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, (adding in firedrake cc again)
On 1 Aug 2019, at 15:43, Madden, Niall <niall.madden@nuigalway.ie> wrote:
Hi Lawrence, Thanks for the swift reply. I don't want to reply to the whole list with too much specific information on the system
Please do reply with the list. That way others who might be able to help will see things.
Can you give us some details on this system? Is it a cray, for example? The system is ICHEC's "kay" - Ireland's national supercomputer for academic purposes: https://www.ichec.ie/about/infrastructure/kay
Normally we find when building on supercomputers (contrary to the one-shot "firedrake-install" unfortunately), that it is best to build PETSc by hand. One then sets the appropriate environment variables and runs firedrake-install --honour-petsc-dir. This might be the best step forward.
Thanks for the tip. I tried compiling PETSc myself:
git clone https://github.com/firedrakeproject/petsc.git (but should I be cloning fromhttps://bitbucket.org/petsc/petsc ?)
No, cloning from the firedrakeproject "fork" is correct. We keep a (reasonably up to date) version of petsc that is known good with all Firedrake components there: it isn't otherwise special.
./configure --with-c2html=0 --download-chaco --download-metis --download-parmetis --download-scalapack --download-hypre --download-mumps --download-netcdf --download-hdf5 --download-pnetcdf --download-exodusii --download-mpich --download-suitesparse=1 --download-petsc4py=1 --download-mpi4py=1 --download-fblaslapack --download-zlib
Then I set PETSC_DIR and PETSC_ARCH, and PYTHONPATH
Then I try to install Firedrake with python3 firedrake-install --no-package-manager -v --disable-ssh --honour-petsc-dir --honour-pythonpath
But it eventually fails compiling libspatialindex for reasons that seem to be related to the mpicxx , but it is a little opaque: undefined reference to `psm2_info_query@PSM2_1.0'
Can you attach the contents of firedrake-install.log (maybe gzip it since it is likely to be large). Thanks, Lawrence
Hi Niall, Koki,
On 2 Aug 2019, at 10:39, Sagiyama, Koki <k.sagiyama@imperial.ac.uk> wrote:
Dear Niall,
It seems to me the following line is critical:
src/dynload.c:5:10: fatal error: Python.h: No such file or directory
You want to have an appropriate package loaded. If you have the `module` command available, you could try `module search python3-dev` or (`module search python-dev`), followed by `module load package_name`.
This certainly looks like an issue. The relevant compile line is: mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DHAVE_DLFCN_H=1 -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c src/dynload.c -o build/temp.linux-x86_64-3.6/src/dynload.o src/dynload.c:5:10: fatal error: Python.h: No such file or directory So it would appear that /usr/include/python3.6m doesn't include the Python header files, can you confirm? In addition to this there is also a link error: /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c _configtest.c -o _configtest.o /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc _configtest.o -L/usr/lib64 -Lbuild/temp.linux-x86_64-3.6 -o _configtest /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0' /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0' collect2: error: ld returned 1 exit status failure. removing: _configtest.c _configtest.o error: Cannot link MPI programs. Check your configuration!!! So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return. And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show
We can also see that you have openmpi loaded (`module list` will show you openmpi), but it is known to cause some issues, so you probably want to unload openmpi and load mpich
This is a bit of red herring I think. If the ICHEC machine suggests using openmpi, then I think that is fine.
Though it is not directly related to the error you are having right now, petsc usually requires some additional parameters when configuring on clusters (--with-batch --known-mpi-shared-libraries=0) (https://www.mcs.anl.gov/petsc/documentation/installation.html ), which we don't see in your previous email.
This is dependent on whether or not the compilation nodes can execute MPI programs, which the ICHEC machine seemingly allows. Cheers, Lawrence
Hi Koki, Lawrence Many thanks for getting back to me.
You want to have an appropriate package loaded. If you have the `module` command available, Yes, I used the Modules package, but python-dev is not avail(able) (because... conda)
So it would appear that /usr/include/python3.6m doesn't include the Python header files, can you confirm? Correct. That dir contains only pyconfig-64.h
Instead I would have to load the conda module, and activate python3. Then /.../packages/conda/2/envs/python3/include/python3.7m/ does indeed have Python.h, and another 100 header files. So, since the docs say "The installation script does not work with anaconda<https://www.continuum.io/downloads> based python installations. This is due to venv issues in anaconda", I am in somewhat of a bind.
So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return. linux-vdso.so.1 => (0x00007ffd5b8f3000) librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007fab64757000) libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007fab64540000) libpsm2.so.2 => /lib64/libpsm2.so.2 (0x00007fab642dd000) librt.so.1 => /lib64/librt.so.1 (0x00007fab640d5000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fab63eb9000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fab63cb5000) libc.so.6 => /lib64/libc.so.6 (0x00007fab638e8000) /lib64/ld-linux-x86-64.so.2 (0x00007fab64c82000) libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007fab6367b000) libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007fab6345a000) libnuma.so.1 => /lib64/libnuma.so.1 (0x00007fab6324e000) libm.so.6 => /lib64/libm.so.6 (0x00007fab62f4c000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fab62d36000)
And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show gcc -I/ichec/packages/openmpi/gcc/3.1.2/include -fexceptions -pthread -L/ichec/packages/libfabric/1.7.1/lib -L/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/libfabric/1.7.1/lib -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/openmpi/gcc/3.1.2/lib -Wl,--enable-new-dtags -L/ichec/packages/openmpi/gcc/3.1.2/lib -lmpi
We can also see that you have openmpi loaded (`module list` will show you openmpi), but it is known to cause some issues, so you probably want to unload openmpi and load mpich This is a bit of red herring I think. If the ICHEC machine suggests using openmpi, then I think that is fine. Good. mpich is not available. Though I could use the Intel compiler, instead of gcc, if there were preferable.
Thanks for the all the (continuing) help guys. Niall. ________________________________ From: Lawrence Mitchell <wence@gmx.li> Sent: Friday 2 August 2019 10:52 To: Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: Madden, Niall <niall.madden@nuigalway.ie>; firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, Koki,
On 2 Aug 2019, at 10:39, Sagiyama, Koki <k.sagiyama@imperial.ac.uk> wrote:
Dear Niall,
It seems to me the following line is critical:
src/dynload.c:5:10: fatal error: Python.h: No such file or directory
You want to have an appropriate package loaded. If you have the `module` command available, you could try `module search python3-dev` or (`module search python-dev`), followed by `module load package_name`.
This certainly looks like an issue. The relevant compile line is: mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DHAVE_DLFCN_H=1 -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c src/dynload.c -o build/temp.linux-x86_64-3.6/src/dynload.o src/dynload.c:5:10: fatal error: Python.h: No such file or directory So it would appear that /usr/include/python3.6m doesn't include the Python header files, can you confirm? In addition to this there is also a link error: /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c _configtest.c -o _configtest.o /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc _configtest.o -L/usr/lib64 -Lbuild/temp.linux-x86_64-3.6 -o _configtest /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0' /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0' collect2: error: ld returned 1 exit status failure. removing: _configtest.c _configtest.o error: Cannot link MPI programs. Check your configuration!!! So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return. And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show
We can also see that you have openmpi loaded (`module list` will show you openmpi), but it is known to cause some issues, so you probably want to unload openmpi and load mpich
This is a bit of red herring I think. If the ICHEC machine suggests using openmpi, then I think that is fine.
Though it is not directly related to the error you are having right now, petsc usually requires some additional parameters when configuring on clusters (--with-batch --known-mpi-shared-libraries=0) (https://www.mcs.anl.gov/petsc/documentation/installation.html ), which we don't see in your previous email.
This is dependent on whether or not the compilation nodes can execute MPI programs, which the ICHEC machine seemingly allows. Cheers, Lawrence
Dear all, Currently it would be safer to say that anaconda is rather untested. I am aware that anaconda venv support has come on and Firedrake has been known to successfully build on anaconda, it’s just a very untested route. You could try to do that. Alternatively, some people who use Firedrake on supercomputers just build Python from source. That’s a fairly straightforward thing to do. For example the script that is used to build Python for Firedrake on ARCHER (the UK national supercomputer) is at: https://github.com/firedrakeproject/firedrake-archer/blob/master/build_pytho... Regards, David From: <firedrake-bounces@imperial.ac.uk> on behalf of "Madden, Niall" <niall.madden@nuigalway.ie> Date: Friday, 2 August 2019 at 12:06 To: Lawrence Mitchell <wence@gmx.li>, "Sagiyama, Koki" <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Koki, Lawrence Many thanks for getting back to me.
You want to have an appropriate package loaded. If you have the `module` command available, Yes, I used the Modules package, but python-dev is not avail(able) (because... conda)
So it would appear that /usr/include/python3.6m doesn't include the Python header files, can you confirm? Correct. That dir contains only pyconfig-64.h
Instead I would have to load the conda module, and activate python3. Then /.../packages/conda/2/envs/python3/include/python3.7m/ does indeed have Python.h, and another 100 header files. So, since the docs say "The installation script does not work with anaconda<https://www.continuum.io/downloads> based python installations. This is due to venv issues in anaconda", I am in somewhat of a bind.
So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return. linux-vdso.so.1 => (0x00007ffd5b8f3000) librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007fab64757000) libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007fab64540000) libpsm2.so.2 => /lib64/libpsm2.so.2 (0x00007fab642dd000) librt.so.1 => /lib64/librt.so.1 (0x00007fab640d5000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fab63eb9000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fab63cb5000) libc.so.6 => /lib64/libc.so.6 (0x00007fab638e8000) /lib64/ld-linux-x86-64.so.2 (0x00007fab64c82000) libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007fab6367b000) libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007fab6345a000) libnuma.so.1 => /lib64/libnuma.so.1 (0x00007fab6324e000) libm.so.6 => /lib64/libm.so.6 (0x00007fab62f4c000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fab62d36000)
And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show gcc -I/ichec/packages/openmpi/gcc/3.1.2/include -fexceptions -pthread -L/ichec/packages/libfabric/1.7.1/lib -L/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/libfabric/1.7.1/lib -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/openmpi/gcc/3.1.2/lib -Wl,--enable-new-dtags -L/ichec/packages/openmpi/gcc/3.1.2/lib -lmpi
We can also see that you have openmpi loaded (`module list` will show you openmpi), but it is known to cause some issues, so you probably want to unload openmpi and load mpich This is a bit of red herring I think. If the ICHEC machine suggests using openmpi, then I think that is fine. Good. mpich is not available. Though I could use the Intel compiler, instead of gcc, if there were preferable.
Thanks for the all the (continuing) help guys. Niall. ________________________________ From: Lawrence Mitchell <wence@gmx.li> Sent: Friday 2 August 2019 10:52 To: Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: Madden, Niall <niall.madden@nuigalway.ie>; firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, Koki,
On 2 Aug 2019, at 10:39, Sagiyama, Koki <k.sagiyama@imperial.ac.uk> wrote:
Dear Niall,
It seems to me the following line is critical:
src/dynload.c:5:10: fatal error: Python.h: No such file or directory
You want to have an appropriate package loaded. If you have the `module` command available, you could try `module search python3-dev` or (`module search python-dev`), followed by `module load package_name`.
This certainly looks like an issue. The relevant compile line is: mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DHAVE_DLFCN_H=1 -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c src/dynload.c -o build/temp.linux-x86_64-3.6/src/dynload.o src/dynload.c:5:10: fatal error: Python.h: No such file or directory So it would appear that /usr/include/python3.6m doesn't include the Python header files, can you confirm? In addition to this there is also a link error: /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c _configtest.c -o _configtest.o /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc _configtest.o -L/usr/lib64 -Lbuild/temp.linux-x86_64-3.6 -o _configtest /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0' /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0' collect2: error: ld returned 1 exit status failure. removing: _configtest.c _configtest.o error: Cannot link MPI programs. Check your configuration!!! So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return. And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show
We can also see that you have openmpi loaded (`module list` will show you openmpi), but it is known to cause some issues, so you probably want to unload openmpi and load mpich
This is a bit of red herring I think. If the ICHEC machine suggests using openmpi, then I think that is fine.
Though it is not directly related to the error you are having right now, petsc usually requires some additional parameters when configuring on clusters (--with-batch --known-mpi-shared-libraries=0) (https://www.mcs.anl.gov/petsc/documentation/installation.html ), which we don't see in your previous email.
This is dependent on whether or not the compilation nodes can execute MPI programs, which the ICHEC machine seemingly allows. Cheers, Lawrence
Hi David, Thanks for the reply. Emboldened by your suggestion that Firedrake and conda might not be so orthogonal after all, I tried that, first rebuilding PETSc (just to be sure). Right now, building Python seems a little daunting. Any way, firedrake-install still failed in the same place, though this time without the complaint about Python.h I attach the latest log file. Again, suggestions welcome. Have a good weekend. Niall. ________________________________ From: Ham, David A <david.ham@imperial.ac.uk> Sent: Friday 2 August 2019 12:12 To: Madden, Niall <niall.madden@nuigalway.ie>; Lawrence Mitchell <wence@gmx.li>; Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Dear all, Currently it would be safer to say that anaconda is rather untested. I am aware that anaconda venv support has come on and Firedrake has been known to successfully build on anaconda, it’s just a very untested route. You could try to do that. Alternatively, some people who use Firedrake on supercomputers just build Python from source. That’s a fairly straightforward thing to do. For example the script that is used to build Python for Firedrake on ARCHER (the UK national supercomputer) is at: https://github.com/firedrakeproject/firedrake-archer/blob/master/build_pytho... Regards, David From: <firedrake-bounces@imperial.ac.uk> on behalf of "Madden, Niall" <niall.madden@nuigalway.ie> Date: Friday, 2 August 2019 at 12:06 To: Lawrence Mitchell <wence@gmx.li>, "Sagiyama, Koki" <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Koki, Lawrence Many thanks for getting back to me.
You want to have an appropriate package loaded.
If you have the `module` command available,
Yes, I used the Modules package, but python-dev is not avail(able) (because... conda)
So it would appear that /usr/include/python3.6m doesn't include
the Python header files, can you confirm?
Correct. That dir contains only pyconfig-64.h Instead I would have to load the conda module, and activate python3. Then /.../packages/conda/2/envs/python3/include/python3.7m/ does indeed have Python.h, and another 100 header files. So, since the docs say "The installation script does not work with anaconda<https://www.continuum.io/downloads> based python installations. This is due to venv issues in anaconda", I am in somewhat of a bind.
So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return.
linux-vdso.so.1 => (0x00007ffd5b8f3000) librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007fab64757000) libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007fab64540000) libpsm2.so.2 => /lib64/libpsm2.so.2 (0x00007fab642dd000) librt.so.1 => /lib64/librt.so.1 (0x00007fab640d5000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fab63eb9000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fab63cb5000) libc.so.6 => /lib64/libc.so.6 (0x00007fab638e8000) /lib64/ld-linux-x86-64.so.2 (0x00007fab64c82000) libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007fab6367b000) libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007fab6345a000) libnuma.so.1 => /lib64/libnuma.so.1 (0x00007fab6324e000) libm.so.6 => /lib64/libm.so.6 (0x00007fab62f4c000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fab62d36000)
And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show gcc -I/ichec/packages/openmpi/gcc/3.1.2/include -fexceptions -pthread -L/ichec/packages/libfabric/1.7.1/lib -L/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/libfabric/1.7.1/lib -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/openmpi/gcc/3.1.2/lib -Wl,--enable-new-dtags -L/ichec/packages/openmpi/gcc/3.1.2/lib -lmpi
We can also see that you have openmpi loaded (`module list` will show you openmpi),
but it is known to cause some issues, so you probably want to unload openmpi and load mpich
This is a bit of red herring I think. If the ICHEC machine suggests using openmpi,
then I think that is fine.
Good. mpich is not available. Though I could use the Intel compiler, instead of gcc, if there were preferable. Thanks for the all the (continuing) help guys. Niall. ________________________________ From: Lawrence Mitchell <wence@gmx.li> Sent: Friday 2 August 2019 10:52 To: Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: Madden, Niall <niall.madden@nuigalway.ie>; firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, Koki,
On 2 Aug 2019, at 10:39, Sagiyama, Koki <k.sagiyama@imperial.ac.uk> wrote:
Dear Niall,
It seems to me the following line is critical:
src/dynload.c:5:10: fatal error: Python.h: No such file or directory
You want to have an appropriate package loaded. If you have the `module` command available, you could try `module search python3-dev` or (`module search python-dev`), followed by `module load package_name`.
This certainly looks like an issue. The relevant compile line is: mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DHAVE_DLFCN_H=1 -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c src/dynload.c -o build/temp.linux-x86_64-3.6/src/dynload.o src/dynload.c:5:10: fatal error: Python.h: No such file or directory So it would appear that /usr/include/python3.6m doesn't include the Python header files, can you confirm? In addition to this there is also a link error: /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c _configtest.c -o _configtest.o /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc _configtest.o -L/usr/lib64 -Lbuild/temp.linux-x86_64-3.6 -o _configtest /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0' /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0' collect2: error: ld returned 1 exit status failure. removing: _configtest.c _configtest.o error: Cannot link MPI programs. Check your configuration!!! So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return. And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show
We can also see that you have openmpi loaded (`module list` will show you openmpi), but it is known to cause some issues, so you probably want to unload openmpi and load mpich
This is a bit of red herring I think. If the ICHEC machine suggests using openmpi, then I think that is fine.
Though it is not directly related to the error you are having right now, petsc usually requires some additional parameters when configuring on clusters (--with-batch --known-mpi-shared-libraries=0) (https://www.mcs.anl.gov/petsc/documentation/installation.html ), which we don't see in your previous email.
This is dependent on whether or not the compilation nodes can execute MPI programs, which the ICHEC machine seemingly allows. Cheers, Lawrence
Hi Niall, So it seems the linker is unable to find function definitions that is most likely in (based on your ldd result): libpsm2.so.2 => /lib64/libpsm2.so.2 (0x00007fab642dd000) I wonder if you need additional packages. Just wondering, does this file exist? Thanks, Koki ________________________________ From: Madden, Niall <niall.madden@nuigalway.ie> Sent: Friday, August 2, 2019 11:52:35 PM To: Ham, David A <david.ham@imperial.ac.uk>; Lawrence Mitchell <wence@gmx.li>; Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi David, Thanks for the reply. Emboldened by your suggestion that Firedrake and conda might not be so orthogonal after all, I tried that, first rebuilding PETSc (just to be sure). Right now, building Python seems a little daunting. Any way, firedrake-install still failed in the same place, though this time without the complaint about Python.h I attach the latest log file. Again, suggestions welcome. Have a good weekend. Niall. ________________________________ From: Ham, David A <david.ham@imperial.ac.uk> Sent: Friday 2 August 2019 12:12 To: Madden, Niall <niall.madden@nuigalway.ie>; Lawrence Mitchell <wence@gmx.li>; Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Dear all, Currently it would be safer to say that anaconda is rather untested. I am aware that anaconda venv support has come on and Firedrake has been known to successfully build on anaconda, it’s just a very untested route. You could try to do that. Alternatively, some people who use Firedrake on supercomputers just build Python from source. That’s a fairly straightforward thing to do. For example the script that is used to build Python for Firedrake on ARCHER (the UK national supercomputer) is at: https://github.com/firedrakeproject/firedrake-archer/blob/master/build_pytho... [https://avatars0.githubusercontent.com/u/5558005?s=400&v=4]<https://github.com/firedrakeproject/firedrake-archer/blob/master/build_python3.7_archer.sh> firedrakeproject/firedrake-archer<https://github.com/firedrakeproject/firedrake-archer/blob/master/build_python3.7_archer.sh> github.com Scripts to build and run Firedrake on Archer, the UK national supercomputer. - firedrakeproject/firedrake-archer Regards, David From: <firedrake-bounces@imperial.ac.uk> on behalf of "Madden, Niall" <niall.madden@nuigalway.ie> Date: Friday, 2 August 2019 at 12:06 To: Lawrence Mitchell <wence@gmx.li>, "Sagiyama, Koki" <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Koki, Lawrence Many thanks for getting back to me.
You want to have an appropriate package loaded.
If you have the `module` command available,
Yes, I used the Modules package, but python-dev is not avail(able) (because... conda)
So it would appear that /usr/include/python3.6m doesn't include
the Python header files, can you confirm?
Correct. That dir contains only pyconfig-64.h Instead I would have to load the conda module, and activate python3. Then /.../packages/conda/2/envs/python3/include/python3.7m/ does indeed have Python.h, and another 100 header files. So, since the docs say "The installation script does not work with anaconda<https://www.continuum.io/downloads> based python installations. This is due to venv issues in anaconda", I am in somewhat of a bind.
So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return.
linux-vdso.so.1 => (0x00007ffd5b8f3000) librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007fab64757000) libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007fab64540000) libpsm2.so.2 => /lib64/libpsm2.so.2 (0x00007fab642dd000) librt.so.1 => /lib64/librt.so.1 (0x00007fab640d5000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fab63eb9000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fab63cb5000) libc.so.6 => /lib64/libc.so.6 (0x00007fab638e8000) /lib64/ld-linux-x86-64.so.2 (0x00007fab64c82000) libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007fab6367b000) libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007fab6345a000) libnuma.so.1 => /lib64/libnuma.so.1 (0x00007fab6324e000) libm.so.6 => /lib64/libm.so.6 (0x00007fab62f4c000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fab62d36000)
And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show gcc -I/ichec/packages/openmpi/gcc/3.1.2/include -fexceptions -pthread -L/ichec/packages/libfabric/1.7.1/lib -L/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/libfabric/1.7.1/lib -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/openmpi/gcc/3.1.2/lib -Wl,--enable-new-dtags -L/ichec/packages/openmpi/gcc/3.1.2/lib -lmpi
We can also see that you have openmpi loaded (`module list` will show you openmpi),
but it is known to cause some issues, so you probably want to unload openmpi and load mpich
This is a bit of red herring I think. If the ICHEC machine suggests using openmpi,
then I think that is fine.
Good. mpich is not available. Though I could use the Intel compiler, instead of gcc, if there were preferable. Thanks for the all the (continuing) help guys. Niall. ________________________________ From: Lawrence Mitchell <wence@gmx.li> Sent: Friday 2 August 2019 10:52 To: Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: Madden, Niall <niall.madden@nuigalway.ie>; firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, Koki,
On 2 Aug 2019, at 10:39, Sagiyama, Koki <k.sagiyama@imperial.ac.uk> wrote:
Dear Niall,
It seems to me the following line is critical:
src/dynload.c:5:10: fatal error: Python.h: No such file or directory
You want to have an appropriate package loaded. If you have the `module` command available, you could try `module search python3-dev` or (`module search python-dev`), followed by `module load package_name`.
This certainly looks like an issue. The relevant compile line is: mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DHAVE_DLFCN_H=1 -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c src/dynload.c -o build/temp.linux-x86_64-3.6/src/dynload.o src/dynload.c:5:10: fatal error: Python.h: No such file or directory So it would appear that /usr/include/python3.6m doesn't include the Python header files, can you confirm? In addition to this there is also a link error: /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c _configtest.c -o _configtest.o /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc _configtest.o -L/usr/lib64 -Lbuild/temp.linux-x86_64-3.6 -o _configtest /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0' /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0' collect2: error: ld returned 1 exit status failure. removing: _configtest.c _configtest.o error: Cannot link MPI programs. Check your configuration!!! So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return. And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show
We can also see that you have openmpi loaded (`module list` will show you openmpi), but it is known to cause some issues, so you probably want to unload openmpi and load mpich
This is a bit of red herring I think. If the ICHEC machine suggests using openmpi, then I think that is fine.
Though it is not directly related to the error you are having right now, petsc usually requires some additional parameters when configuring on clusters (--with-batch --known-mpi-shared-libraries=0) (https://www.mcs.anl.gov/petsc/documentation/installation.html ), which we don't see in your previous email.
This is dependent on whether or not the compilation nodes can execute MPI programs, which the ICHEC machine seemingly allows. Cheers, Lawrence
To be a bit more expansive, mpi4py is failing to build because it can’t find the symbols that koki points to: building 'mpi4py.dl' extension mpicc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DHAVE_DLFCN_H=1 -I/ichec/home/users/whoever/r/libs/firedrake/include -I/ichec/packages/conda/2/envs/python3/include/python3.7m -c src/dynload.c -o build/temp.linux-x86_64-3.7/src/dynload.o gcc -pthread -shared -B /ichec/packages/conda/2/envs/python3/compiler_compat -L/ichec/packages/conda/2/envs/python3/lib -Wl,-rpath=/ichec/packages/conda/2/envs/python3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/src/dynload.o -Lbuild/temp.linux-x86_64-3.7 -o build/lib.linux-x86_64-3.7/mpi4py/dl.cpython-37m-x86_64-linux-gnu.so checking for MPI compile and link ... /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/ichec/home/users/whoever/r/libs/firedrake/include -I/ichec/packages/conda/2/envs/python3/include/python3.7m -c _configtest.c -o _configtest.o success! removing: _configtest.c _configtest.o /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/ichec/home/users/whoever/r/libs/firedrake/include -I/ichec/packages/conda/2/envs/python3/include/python3.7m -c _configtest.c -o _configtest.o /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.7 -o _configtest /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0' /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0' collect2: error: ld returned 1 exit status failure. removing: _configtest.c _configtest.o error: Cannot link MPI programs. Check your configuration!!! You could try building mpi4py in a clean venv. If this fails in the same way then it will produce a relatively small problem which you can take to the cluster admins. Regards, David From: "Sagiyama, Koki" <k.sagiyama@imperial.ac.uk> Date: Tuesday, 6 August 2019 at 10:31 To: "Madden, Niall" <niall.madden@nuigalway.ie>, "Ham, David A" <david.ham@imperial.ac.uk>, Lawrence Mitchell <wence@gmx.li> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, So it seems the linker is unable to find function definitions that is most likely in (based on your ldd result): libpsm2.so.2 => /lib64/libpsm2.so.2 (0x00007fab642dd000) I wonder if you need additional packages. Just wondering, does this file exist? Thanks, Koki ________________________________ From: Madden, Niall <niall.madden@nuigalway.ie> Sent: Friday, August 2, 2019 11:52:35 PM To: Ham, David A <david.ham@imperial.ac.uk>; Lawrence Mitchell <wence@gmx.li>; Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi David, Thanks for the reply. Emboldened by your suggestion that Firedrake and conda might not be so orthogonal after all, I tried that, first rebuilding PETSc (just to be sure). Right now, building Python seems a little daunting. Any way, firedrake-install still failed in the same place, though this time without the complaint about Python.h I attach the latest log file. Again, suggestions welcome. Have a good weekend. Niall. ________________________________ From: Ham, David A <david.ham@imperial.ac.uk> Sent: Friday 2 August 2019 12:12 To: Madden, Niall <niall.madden@nuigalway.ie>; Lawrence Mitchell <wence@gmx.li>; Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Dear all, Currently it would be safer to say that anaconda is rather untested. I am aware that anaconda venv support has come on and Firedrake has been known to successfully build on anaconda, it’s just a very untested route. You could try to do that. Alternatively, some people who use Firedrake on supercomputers just build Python from source. That’s a fairly straightforward thing to do. For example the script that is used to build Python for Firedrake on ARCHER (the UK national supercomputer) is at: https://github.com/firedrakeproject/firedrake-archer/blob/master/build_pytho... [Image removed by sender.]<https://github.com/firedrakeproject/firedrake-archer/blob/master/build_python3.7_archer.sh> firedrakeproject/firedrake-archer<https://github.com/firedrakeproject/firedrake-archer/blob/master/build_python3.7_archer.sh> github.com Scripts to build and run Firedrake on Archer, the UK national supercomputer. - firedrakeproject/firedrake-archer Regards, David From: <firedrake-bounces@imperial.ac.uk> on behalf of "Madden, Niall" <niall.madden@nuigalway.ie> Date: Friday, 2 August 2019 at 12:06 To: Lawrence Mitchell <wence@gmx.li>, "Sagiyama, Koki" <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Koki, Lawrence Many thanks for getting back to me.
You want to have an appropriate package loaded.
If you have the `module` command available,
Yes, I used the Modules package, but python-dev is not avail(able) (because... conda)
So it would appear that /usr/include/python3.6m doesn't include
the Python header files, can you confirm?
Correct. That dir contains only pyconfig-64.h Instead I would have to load the conda module, and activate python3. Then /.../packages/conda/2/envs/python3/include/python3.7m/ does indeed have Python.h, and another 100 header files. So, since the docs say "The installation script does not work with anaconda<https://www.continuum.io/downloads> based python installations. This is due to venv issues in anaconda", I am in somewhat of a bind.
So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return.
linux-vdso.so.1 => (0x00007ffd5b8f3000) librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007fab64757000) libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007fab64540000) libpsm2.so.2 => /lib64/libpsm2.so.2 (0x00007fab642dd000) librt.so.1 => /lib64/librt.so.1 (0x00007fab640d5000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fab63eb9000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fab63cb5000) libc.so.6 => /lib64/libc.so.6 (0x00007fab638e8000) /lib64/ld-linux-x86-64.so.2 (0x00007fab64c82000) libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007fab6367b000) libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007fab6345a000) libnuma.so.1 => /lib64/libnuma.so.1 (0x00007fab6324e000) libm.so.6 => /lib64/libm.so.6 (0x00007fab62f4c000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fab62d36000)
And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show gcc -I/ichec/packages/openmpi/gcc/3.1.2/include -fexceptions -pthread -L/ichec/packages/libfabric/1.7.1/lib -L/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/libfabric/1.7.1/lib -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/openmpi/gcc/3.1.2/lib -Wl,--enable-new-dtags -L/ichec/packages/openmpi/gcc/3.1.2/lib -lmpi
We can also see that you have openmpi loaded (`module list` will show you openmpi),
but it is known to cause some issues, so you probably want to unload openmpi and load mpich
This is a bit of red herring I think. If the ICHEC machine suggests using openmpi,
then I think that is fine.
Good. mpich is not available. Though I could use the Intel compiler, instead of gcc, if there were preferable. Thanks for the all the (continuing) help guys. Niall. ________________________________ From: Lawrence Mitchell <wence@gmx.li> Sent: Friday 2 August 2019 10:52 To: Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: Madden, Niall <niall.madden@nuigalway.ie>; firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, Koki,
On 2 Aug 2019, at 10:39, Sagiyama, Koki <k.sagiyama@imperial.ac.uk> wrote:
Dear Niall,
It seems to me the following line is critical:
src/dynload.c:5:10: fatal error: Python.h: No such file or directory
You want to have an appropriate package loaded. If you have the `module` command available, you could try `module search python3-dev` or (`module search python-dev`), followed by `module load package_name`.
This certainly looks like an issue. The relevant compile line is: mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DHAVE_DLFCN_H=1 -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c src/dynload.c -o build/temp.linux-x86_64-3.6/src/dynload.o src/dynload.c:5:10: fatal error: Python.h: No such file or directory So it would appear that /usr/include/python3.6m doesn't include the Python header files, can you confirm? In addition to this there is also a link error: /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c _configtest.c -o _configtest.o /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc _configtest.o -L/usr/lib64 -Lbuild/temp.linux-x86_64-3.6 -o _configtest /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0' /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0' collect2: error: ld returned 1 exit status failure. removing: _configtest.c _configtest.o error: Cannot link MPI programs. Check your configuration!!! So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return. And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show
We can also see that you have openmpi loaded (`module list` will show you openmpi), but it is known to cause some issues, so you probably want to unload openmpi and load mpich
This is a bit of red herring I think. If the ICHEC machine suggests using openmpi, then I think that is fine.
Though it is not directly related to the error you are having right now, petsc usually requires some additional parameters when configuring on clusters (--with-batch --known-mpi-shared-libraries=0) (https://www.mcs.anl.gov/petsc/documentation/installation.html ), which we don't see in your previous email.
This is dependent on whether or not the compilation nodes can execute MPI programs, which the ICHEC machine seemingly allows. Cheers, Lawrence
Hi all, I've returned to this task, and tried to follow David's suggestion of building on a clean conda environment. This time, I used the intel compiler (mpiicc). Trying to follow the steps in the install script, we cloned firedrake, ufl, fiat, PyOP2, COFFEE, loopy, petsc, supermesh, spatialindex, etc. Then we can successfully install all these. In particular, we can configure/make/make install petsc Finally, we run, in firedrake, python setup.py install with LDSHARED, CFLAGS, CC, CXX, and prefix set appropriately. This **seems** to work OK, and culminates in the output running install_egg_info Writing /MYOB/.conda/envs/firedrake/lib/python3.6/site-packages /firedrake-0.13.0_2890.g8f880fd3-py3.6.egg-info But I am at a loss as to how to complete the installation. (My fuzzy understanding of conda and venvs does not help). That is, if I try echo "import firedrake" | python I get Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/MYOB/r/libs/firedrake/firedrake/__init__.py", line 5, in <module> if "PETSC_DIR" in os.environ and not config["options"]["honour_petsc_dir"]: TypeError: 'NoneType' object is not subscriptable Unsetting PETSC_DIR does not help (note change in error): Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/MYOB/r/libs/firedrake/firedrake/__init__.py", line 8, in <module> elif "PETSC_DIR" not in os.environ and config["options"]["honour_petsc_dir"]: TypeError: 'NoneType' object is not subscriptable I note that, the "activate" script is not in firedrake/bin Neither, as it happens, is firedrake-update, but firedrake-clean, firedrake-install, and firedrake-zenodo are. Suggestions most welcome... Niall. -- ________________________________ From: Ham, David A <david.ham@imperial.ac.uk> Sent: Tuesday 6 August 2019 11:33 To: Sagiyama, Koki <k.sagiyama@imperial.ac.uk>; Madden, Niall <niall.madden@nuigalway.ie>; Lawrence Mitchell <wence@gmx.li> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager To be a bit more expansive, mpi4py is failing to build because it can’t find the symbols that koki points to: building 'mpi4py.dl' extension mpicc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DHAVE_DLFCN_H=1 -I/ichec/home/users/whoever/r/libs/firedrake/include -I/ichec/packages/conda/2/envs/python3/include/python3.7m -c src/dynload.c -o build/temp.linux-x86_64-3.7/src/dynload.o gcc -pthread -shared -B /ichec/packages/conda/2/envs/python3/compiler_compat -L/ichec/packages/conda/2/envs/python3/lib -Wl,-rpath=/ichec/packages/conda/2/envs/python3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/src/dynload.o -Lbuild/temp.linux-x86_64-3.7 -o build/lib.linux-x86_64-3.7/mpi4py/dl.cpython-37m-x86_64-linux-gnu.so checking for MPI compile and link ... /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/ichec/home/users/whoever/r/libs/firedrake/include -I/ichec/packages/conda/2/envs/python3/include/python3.7m -c _configtest.c -o _configtest.o success! removing: _configtest.c _configtest.o /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/ichec/home/users/whoever/r/libs/firedrake/include -I/ichec/packages/conda/2/envs/python3/include/python3.7m -c _configtest.c -o _configtest.o /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.7 -o _configtest /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0' /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0' collect2: error: ld returned 1 exit status failure. removing: _configtest.c _configtest.o error: Cannot link MPI programs. Check your configuration!!! You could try building mpi4py in a clean venv. If this fails in the same way then it will produce a relatively small problem which you can take to the cluster admins. Regards, David From: "Sagiyama, Koki" <k.sagiyama@imperial.ac.uk> Date: Tuesday, 6 August 2019 at 10:31 To: "Madden, Niall" <niall.madden@nuigalway.ie>, "Ham, David A" <david.ham@imperial.ac.uk>, Lawrence Mitchell <wence@gmx.li> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, So it seems the linker is unable to find function definitions that is most likely in (based on your ldd result): libpsm2.so.2 => /lib64/libpsm2.so.2 (0x00007fab642dd000) I wonder if you need additional packages. Just wondering, does this file exist? Thanks, Koki ________________________________ From: Madden, Niall <niall.madden@nuigalway.ie> Sent: Friday, August 2, 2019 11:52:35 PM To: Ham, David A <david.ham@imperial.ac.uk>; Lawrence Mitchell <wence@gmx.li>; Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi David, Thanks for the reply. Emboldened by your suggestion that Firedrake and conda might not be so orthogonal after all, I tried that, first rebuilding PETSc (just to be sure). Right now, building Python seems a little daunting. Any way, firedrake-install still failed in the same place, though this time without the complaint about Python.h I attach the latest log file. Again, suggestions welcome. Have a good weekend. Niall. ________________________________ From: Ham, David A <david.ham@imperial.ac.uk> Sent: Friday 2 August 2019 12:12 To: Madden, Niall <niall.madden@nuigalway.ie>; Lawrence Mitchell <wence@gmx.li>; Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Dear all, Currently it would be safer to say that anaconda is rather untested. I am aware that anaconda venv support has come on and Firedrake has been known to successfully build on anaconda, it’s just a very untested route. You could try to do that. Alternatively, some people who use Firedrake on supercomputers just build Python from source. That’s a fairly straightforward thing to do. For example the script that is used to build Python for Firedrake on ARCHER (the UK national supercomputer) is at: https://github.com/firedrakeproject/firedrake-archer/blob/master/build_pytho... [Image removed by sender.]<https://github.com/firedrakeproject/firedrake-archer/blob/master/build_python3.7_archer.sh> firedrakeproject/firedrake-archer<https://github.com/firedrakeproject/firedrake-archer/blob/master/build_python3.7_archer.sh> github.com Scripts to build and run Firedrake on Archer, the UK national supercomputer. - firedrakeproject/firedrake-archer Regards, David From: <firedrake-bounces@imperial.ac.uk> on behalf of "Madden, Niall" <niall.madden@nuigalway.ie> Date: Friday, 2 August 2019 at 12:06 To: Lawrence Mitchell <wence@gmx.li>, "Sagiyama, Koki" <k.sagiyama@imperial.ac.uk> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Koki, Lawrence Many thanks for getting back to me.
You want to have an appropriate package loaded.
If you have the `module` command available,
Yes, I used the Modules package, but python-dev is not avail(able) (because... conda)
So it would appear that /usr/include/python3.6m doesn't include
the Python header files, can you confirm?
Correct. That dir contains only pyconfig-64.h Instead I would have to load the conda module, and activate python3. Then /.../packages/conda/2/envs/python3/include/python3.7m/ does indeed have Python.h, and another 100 header files. So, since the docs say "The installation script does not work with anaconda<https://www.continuum.io/downloads> based python installations. This is due to venv issues in anaconda", I am in somewhat of a bind.
So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return.
linux-vdso.so.1 => (0x00007ffd5b8f3000) librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007fab64757000) libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007fab64540000) libpsm2.so.2 => /lib64/libpsm2.so.2 (0x00007fab642dd000) librt.so.1 => /lib64/librt.so.1 (0x00007fab640d5000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fab63eb9000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fab63cb5000) libc.so.6 => /lib64/libc.so.6 (0x00007fab638e8000) /lib64/ld-linux-x86-64.so.2 (0x00007fab64c82000) libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007fab6367b000) libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007fab6345a000) libnuma.so.1 => /lib64/libnuma.so.1 (0x00007fab6324e000) libm.so.6 => /lib64/libm.so.6 (0x00007fab62f4c000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fab62d36000)
And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show gcc -I/ichec/packages/openmpi/gcc/3.1.2/include -fexceptions -pthread -L/ichec/packages/libfabric/1.7.1/lib -L/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/libfabric/1.7.1/lib -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/ichec/packages/openmpi/gcc/3.1.2/lib -Wl,--enable-new-dtags -L/ichec/packages/openmpi/gcc/3.1.2/lib -lmpi
We can also see that you have openmpi loaded (`module list` will show you openmpi),
but it is known to cause some issues, so you probably want to unload openmpi and load mpich
This is a bit of red herring I think. If the ICHEC machine suggests using openmpi,
then I think that is fine.
Good. mpich is not available. Though I could use the Intel compiler, instead of gcc, if there were preferable. Thanks for the all the (continuing) help guys. Niall. ________________________________ From: Lawrence Mitchell <wence@gmx.li> Sent: Friday 2 August 2019 10:52 To: Sagiyama, Koki <k.sagiyama@imperial.ac.uk> Cc: Madden, Niall <niall.madden@nuigalway.ie>; firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, Koki,
On 2 Aug 2019, at 10:39, Sagiyama, Koki <k.sagiyama@imperial.ac.uk> wrote:
Dear Niall,
It seems to me the following line is critical:
src/dynload.c:5:10: fatal error: Python.h: No such file or directory
You want to have an appropriate package loaded. If you have the `module` command available, you could try `module search python3-dev` or (`module search python-dev`), followed by `module load package_name`.
This certainly looks like an issue. The relevant compile line is: mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DHAVE_DLFCN_H=1 -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c src/dynload.c -o build/temp.linux-x86_64-3.6/src/dynload.o src/dynload.c:5:10: fatal error: Python.h: No such file or directory So it would appear that /usr/include/python3.6m doesn't include the Python header files, can you confirm? In addition to this there is also a link error: /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/MYOB/r/libs/firedrake/include -I/usr/include/python3.6m -c _configtest.c -o _configtest.o /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc _configtest.o -L/usr/lib64 -Lbuild/temp.linux-x86_64-3.6 -o _configtest /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0' /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0' collect2: error: ld returned 1 exit status failure. removing: _configtest.c _configtest.o error: Cannot link MPI programs. Check your configuration!!! So I wonder if there is a module that should be loaded but isn't. What does: ldd /ichec/packages/libfabric/1.7.1/lib/libfabric.so.1 return. And also /ichec/packages/openmpi/gcc/3.1.2/bin/mpicc -show
We can also see that you have openmpi loaded (`module list` will show you openmpi), but it is known to cause some issues, so you probably want to unload openmpi and load mpich
This is a bit of red herring I think. If the ICHEC machine suggests using openmpi, then I think that is fine.
Though it is not directly related to the error you are having right now, petsc usually requires some additional parameters when configuring on clusters (--with-batch --known-mpi-shared-libraries=0) (https://www.mcs.anl.gov/petsc/documentation/installation.html ), which we don't see in your previous email.
This is dependent on whether or not the compilation nodes can execute MPI programs, which the ICHEC machine seemingly allows. Cheers, Lawrence
Hi Niall, On Wed, 21 Aug 2019 at 15:04, Madden, Niall <niall.madden@nuigalway.ie> wrote:
Hi all, I've returned to this task, and tried to follow David's suggestion of building on a clean conda environment. This time, I used the intel compiler (mpiicc).
Trying to follow the steps in the install script, we cloned firedrake, ufl, fiat, PyOP2, COFFEE, loopy, petsc, supermesh, spatialindex, etc.
Then we can successfully install all these. In particular, we can configure/make/make install petsc
Finally, we run, in firedrake, python setup.py install with LDSHARED, CFLAGS, CC, CXX, and prefix set appropriately.
[...]
That is, if I try echo "import firedrake" | python I get
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/MYOB/r/libs/firedrake/firedrake/__init__.py", line 5, in <module> if "PETSC_DIR" in os.environ and not config["options"]["honour_petsc_dir"]: TypeError: 'NoneType' object is not subscriptable
[...]
I note that, the "activate" script is not in firedrake/bin Neither, as it happens, is firedrake-update, but firedrake-clean, firedrake-install, and firedrake-zenodo are.
This is an idiosyncrasy of the firedrake-install process. "firedrake-update" is actually just a copy of "firedrake-install" with a different name (so it knows it's in update mode). firedrake-{clean/install/zenodo} are created in the bin directory by setup.py, but firedrake-update is copied _by the firedrake-install process_. Hence it not existing because you did everything by hand. You can "fix" this by just copying firedrake-install to firedrake-update in the bin directory. The issue with the TypeError above is related. As well as creating the firedrake-update script, firedrake-install also writes a little configuration file that remembers the options you installed firedrake with (so that firedrake-update maintains them). Again, because you didn't run firedrake-install, this file doesn't exist. The relevant file to be created is `configuration.json` in the firedrake_configuration subdirectory of the firedrake source tree. Mine looks like: {"options": {"package_manager": false, "minimal_petsc": false, "mpicc": null, "mpicxx": null, "mpif90": null, "mpiexec": null, "disable_ssh": false, "honour_petsc_dir": true, "show_petsc_configure_options": false, "slepc": true, "packages": [], "honour_pythonpath": false, "petsc_int_type": "int32", "cache_dir": "/home/vtdb72/Documents/work/src/firedrake/.cache"}, "environment": {}, "additions": []} Editing this by hand is, however, rather error prone. Additionally, I'm not necessarily sure that firedrake-update will run as normal. Given you started with a clean conda environment, did you run into issues just running firedrake-install inside that? Thanks, Lawrence
Hi Niall, One more piece of the puzzle for you: the activate script is part of Python venv. One of the first things that the install script does is create a Firedrake venv into which Firedrake is installed. It is the creation of the venv which creates the activate script. I would reiterate what Lawrence says: installing by hand like this is very, very error-prone. It would almost certainly be better to run the install script. In particular, installing by hand and then hoping that the update script will work sounds very optimistic indeed. Regards, David On 21/08/2019, 16:22, "Lawrence Mitchell" <wence@gmx.li> wrote: Hi Niall, On Wed, 21 Aug 2019 at 15:04, Madden, Niall <niall.madden@nuigalway.ie> wrote: > > Hi all, > I've returned to this task, and tried to follow David's suggestion > of building on a clean conda environment. This time, I used the intel compiler (mpiicc). > > Trying to follow the steps in the install script, we cloned > firedrake, ufl, fiat, PyOP2, COFFEE, loopy, petsc, supermesh, spatialindex, etc. > > Then we can successfully install all these. In particular, we can > configure/make/make install petsc > > Finally, we run, in firedrake, > python setup.py install > with LDSHARED, CFLAGS, CC, CXX, and prefix set appropriately. [...] > That is, if I try > echo "import firedrake" | python > I get > > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "/MYOB/r/libs/firedrake/firedrake/__init__.py", line 5, in <module> > if "PETSC_DIR" in os.environ and not config["options"]["honour_petsc_dir"]: > TypeError: 'NoneType' object is not subscriptable [...] > I note that, the "activate" script is not in firedrake/bin > Neither, as it happens, is firedrake-update, but firedrake-clean, firedrake-install, and firedrake-zenodo are. This is an idiosyncrasy of the firedrake-install process. "firedrake-update" is actually just a copy of "firedrake-install" with a different name (so it knows it's in update mode). firedrake-{clean/install/zenodo} are created in the bin directory by setup.py, but firedrake-update is copied _by the firedrake-install process_. Hence it not existing because you did everything by hand. You can "fix" this by just copying firedrake-install to firedrake-update in the bin directory. The issue with the TypeError above is related. As well as creating the firedrake-update script, firedrake-install also writes a little configuration file that remembers the options you installed firedrake with (so that firedrake-update maintains them). Again, because you didn't run firedrake-install, this file doesn't exist. The relevant file to be created is `configuration.json` in the firedrake_configuration subdirectory of the firedrake source tree. Mine looks like: {"options": {"package_manager": false, "minimal_petsc": false, "mpicc": null, "mpicxx": null, "mpif90": null, "mpiexec": null, "disable_ssh": false, "honour_petsc_dir": true, "show_petsc_configure_options": false, "slepc": true, "packages": [], "honour_pythonpath": false, "petsc_int_type": "int32", "cache_dir": "/home/vtdb72/Documents/work/src/firedrake/.cache"}, "environment": {}, "additions": []} Editing this by hand is, however, rather error prone. Additionally, I'm not necessarily sure that firedrake-update will run as normal. Given you started with a clean conda environment, did you run into issues just running firedrake-install inside that? Thanks, Lawrence
Hi guys, Thanks for the replies. Lawrence:
Given you started with a clean conda environment, did you run into issues just running firedrake-install inside that? Yes. Fails when compiling Petsc. Since I can compile Petsc by hand, I tried that passed --honour-petsc-dir when trying to build firedrake. But then it fails to compile h5py The log reports "/MYOB/firedrake/src/h5py/h5py/api_compat.h(27): catastrophic error: cannot open source file "hdf5.h""
David:
One more piece of the puzzle for you: the activate script is part of Python venv. One of the first things that the install script does is create a Firedrake venv into which Firedrake is installed. It is the creation of the venv which creates the activate script. I see. Thanks.
I would reiterate what Lawrence says: installing by hand like this is very, very error-prone. So too is not installing by hand it seems. At least for my setup.
In particular, installing by hand and then hoping that the update script will work sounds very optimistic indeed. I share your pessimism. I hadn't expressed that hope, I just noted that that file hadn't been created.
I guess I could try building Python, as David suggested earlier. (Or going back to FEniCS, which installed without problems - but I do like quads :-) Niall. ________________________________ From: Ham, David A <david.ham@imperial.ac.uk> Sent: Thursday 22 August 2019 08:48 To: Lawrence Mitchell <wence@gmx.li>; Madden, Niall <niall.madden@nuigalway.ie> Cc: Sagiyama, Koki <k.sagiyama@imperial.ac.uk>; firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, One more piece of the puzzle for you: the activate script is part of Python venv. One of the first things that the install script does is create a Firedrake venv into which Firedrake is installed. It is the creation of the venv which creates the activate script. I would reiterate what Lawrence says: installing by hand like this is very, very error-prone. It would almost certainly be better to run the install script. In particular, installing by hand and then hoping that the update script will work sounds very optimistic indeed. Regards, David On 21/08/2019, 16:22, "Lawrence Mitchell" <wence@gmx.li> wrote: Hi Niall, On Wed, 21 Aug 2019 at 15:04, Madden, Niall <niall.madden@nuigalway.ie> wrote: > > Hi all, > I've returned to this task, and tried to follow David's suggestion > of building on a clean conda environment. This time, I used the intel compiler (mpiicc). > > Trying to follow the steps in the install script, we cloned > firedrake, ufl, fiat, PyOP2, COFFEE, loopy, petsc, supermesh, spatialindex, etc. > > Then we can successfully install all these. In particular, we can > configure/make/make install petsc > > Finally, we run, in firedrake, > python setup.py install > with LDSHARED, CFLAGS, CC, CXX, and prefix set appropriately. [...] > That is, if I try > echo "import firedrake" | python > I get > > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "/MYOB/r/libs/firedrake/firedrake/__init__.py", line 5, in <module> > if "PETSC_DIR" in os.environ and not config["options"]["honour_petsc_dir"]: > TypeError: 'NoneType' object is not subscriptable [...] > I note that, the "activate" script is not in firedrake/bin > Neither, as it happens, is firedrake-update, but firedrake-clean, firedrake-install, and firedrake-zenodo are. This is an idiosyncrasy of the firedrake-install process. "firedrake-update" is actually just a copy of "firedrake-install" with a different name (so it knows it's in update mode). firedrake-{clean/install/zenodo} are created in the bin directory by setup.py, but firedrake-update is copied _by the firedrake-install process_. Hence it not existing because you did everything by hand. You can "fix" this by just copying firedrake-install to firedrake-update in the bin directory. The issue with the TypeError above is related. As well as creating the firedrake-update script, firedrake-install also writes a little configuration file that remembers the options you installed firedrake with (so that firedrake-update maintains them). Again, because you didn't run firedrake-install, this file doesn't exist. The relevant file to be created is `configuration.json` in the firedrake_configuration subdirectory of the firedrake source tree. Mine looks like: {"options": {"package_manager": false, "minimal_petsc": false, "mpicc": null, "mpicxx": null, "mpif90": null, "mpiexec": null, "disable_ssh": false, "honour_petsc_dir": true, "show_petsc_configure_options": false, "slepc": true, "packages": [], "honour_pythonpath": false, "petsc_int_type": "int32", "cache_dir": "/home/vtdb72/Documents/work/src/firedrake/.cache"}, "environment": {}, "additions": []} Editing this by hand is, however, rather error prone. Additionally, I'm not necessarily sure that firedrake-update will run as normal. Given you started with a clean conda environment, did you run into issues just running firedrake-install inside that? Thanks, Lawrence
To be honest, building Python from source is far easier than manually installing Firedrake. "Madden, Niall" <niall.madden@nuigalway.ie> writes:
I would reiterate what Lawrence says: installing by hand like this is very, very error-prone. So too is not installing by hand it seems. At least for my setup.
In particular, installing by hand and then hoping that the update script will work sounds very optimistic indeed. I share your pessimism. I hadn't expressed that hope, I just noted that that file hadn't been created.
I guess I could try building Python, as David suggested earlier. (Or going back to FEniCS, which installed without problems - but I do like quads :-)
Niall.
Hi Miklós, David, Lawrence,
To be honest, building Python from source is far easier than manually installing Firedrake. OK. I tried that. Didn't work.
When you compiled PETSc by hand, which configure options
did you use? We depend on PETSc having downloaded quite a few
things for us, in particular including hdf5 . It looks like you didn’t
build PETSc with that option.... --show-petsc-configure-options
You are right, David: I was relying on a system installed hdf5, which was inforced when making Petsc with --with-hdf5-dir=... It is unclear to me how to tell firedrake install to use that hdf5 I'll try making Petsc without that later... By the way, "--show-petsc-configure-options" does not seem to work when PETSC_DIR is set. Instead it complains that I need --honour-petsc-dir Then it complains about the lack of package manager. But python3 firedrake-install --show-petsc-configure-options --honour-petsc-dir --no-package-manager just tries to install firedrake without reporting the Petsc options. Have a good weekend, Niall. Dr Niall Madden, School of Mathematics, Statistics and Applied Mathematics, NUI Galway Vice-dean for Equality, Diversity and Inclusion, College of Science Web: http://www.maths.nuigalway.ie/~niall Local co-organiser: 23rd Conference of the International Linear Algebra Society - www.ilas2020.ie<http://ilas2020.ie/> Associate Editor of Numerical Algorithms (Springer) ________________________________ From: Miklós Homolya <miklos.homolya@tum.de> Sent: Friday 23 August 2019 05:09 To: Madden, Niall <niall.madden@nuigalway.ie> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager To be honest, building Python from source is far easier than manually installing Firedrake. "Madden, Niall" <niall.madden@nuigalway.ie> writes:
I would reiterate what Lawrence says: installing by hand like this is very, very error-prone. So too is not installing by hand it seems. At least for my setup.
In particular, installing by hand and then hoping that the update script will work sounds very optimistic indeed. I share your pessimism. I hadn't expressed that hope, I just noted that that file hadn't been created.
I guess I could try building Python, as David suggested earlier. (Or going back to FEniCS, which installed without problems - but I do like quads :-)
Niall.
Hi Niall, apologies that this is still not working, although we would like to get to the bottom.
On 23 Aug 2019, at 16:07, Madden, Niall <niall.madden@nuigalway.ie> wrote:
Hi Miklós, David, Lawrence,
To be honest, building Python from source is far easier than manually installing Firedrake. OK. I tried that. Didn't work.
Can you explain how things failed? Building Python didn't work, or that worked, and then firedrake-install failed?
When you compiled PETSc by hand, which configure options did you use? We depend on PETSc having downloaded quite a few things for us, in particular including hdf5 . It looks like you didn’t build PETSc with that option.... --show-petsc-configure-options You are right, David: I was relying on a system installed hdf5, which was inforced when making Petsc with --with-hdf5-dir=... It is unclear to me how to tell firedrake install to use that hdf5
I'll try making Petsc without that later...
Our issue is that we basically rely on PETSc to build most of the compiled dependencies, and hence if PETSc was built with --with-hdf5-dir=... (rather than --download-hdf5) we don't find it. I think this is actually easy to override: we only need to interpose in one place. Can you try using the firedrake-install script from here: https://raw.githubusercontent.com/firedrakeproject/firedrake/wence/fix/insta... You can then do: export HDF5_DIR=/path/you/used/for/--with-hdf5-dir python3 firedrake-install --honour-petsc-dir And hopefully this will pick up the right HDF5.
By the way, "--show-petsc-configure-options" does not seem to work when PETSC_DIR is set. Instead it complains that I need --honour-petsc-dir Then it complains about the lack of package manager.
But python3 firedrake-install --show-petsc-configure-options --honour-petsc-dir --no-package-manager just tries to install firedrake without reporting the Petsc options.
Apologies for this. I just pushed a fix for this logic, so if you grab a new version of the firedrake-install script, it should work with just "python3 firedrake-install --show-petsc-configure-options" (the above linked script also includes this change). Our major issue with installation is that we want a unified solution independent of platform. Since package managers are mostly platform-specific, this is somewhat challenging. There are some possible cross-platform package management systems, but I do not yet believe that they are suitable for anything other than expert users. My real hope is that proper containerised solutions will appear on HPC systems in the next year or so, which will simplify all of this significantly. Thanks, Lawrence
On Fri, Aug 23, 2019 at 11:28 AM Lawrence Mitchell <wence@gmx.li> wrote:
Hi Niall,
apologies that this is still not working, although we would like to get to the bottom.
On 23 Aug 2019, at 16:07, Madden, Niall <niall.madden@nuigalway.ie> wrote:
Hi Miklós, David, Lawrence,
To be honest, building Python from source is far easier than manually installing Firedrake. OK. I tried that. Didn't work.
Can you explain how things failed? Building Python didn't work, or that worked, and then firedrake-install failed?
When you compiled PETSc by hand, which configure options did you use? We depend on PETSc having downloaded quite a few things for us, in particular including hdf5 . It looks like you didn’t build PETSc with that option.... --show-petsc-configure-options You are right, David: I was relying on a system installed hdf5, which was inforced when making Petsc with --with-hdf5-dir=... It is unclear to me how to tell firedrake install to use that hdf5
I'll try making Petsc without that later...
Our issue is that we basically rely on PETSc to build most of the compiled dependencies, and hence if PETSc was built with --with-hdf5-dir=... (rather than --download-hdf5) we don't find it.
Why doesn't this work? PETSc knows where HDF5 is, no matter how it is specified, and you can query it from Python. Thanks, Matt
I think this is actually easy to override: we only need to interpose in one place.
Can you try using the firedrake-install script from here: https://raw.githubusercontent.com/firedrakeproject/firedrake/wence/fix/insta...
You can then do:
export HDF5_DIR=/path/you/used/for/--with-hdf5-dir
python3 firedrake-install --honour-petsc-dir
And hopefully this will pick up the right HDF5.
By the way, "--show-petsc-configure-options" does not seem to work when PETSC_DIR is set. Instead it complains that I need --honour-petsc-dir Then it complains about the lack of package manager.
But python3 firedrake-install --show-petsc-configure-options --honour-petsc-dir --no-package-manager just tries to install firedrake without reporting the Petsc options.
Apologies for this. I just pushed a fix for this logic, so if you grab a new version of the firedrake-install script, it should work with just "python3 firedrake-install --show-petsc-configure-options" (the above linked script also includes this change).
Our major issue with installation is that we want a unified solution independent of platform. Since package managers are mostly platform-specific, this is somewhat challenging. There are some possible cross-platform package management systems, but I do not yet believe that they are suitable for anything other than expert users.
My real hope is that proper containerised solutions will appear on HPC systems in the next year or so, which will simplify all of this significantly.
Thanks,
Lawrence _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
On 23 Aug 2019, at 17:03, Matthew Knepley <knepley@gmail.com> wrote:
[...]
Our issue is that we basically rely on PETSc to build most of the compiled dependencies, and hence if PETSc was built with --with-hdf5-dir=... (rather than --download-hdf5) we don't find it.
Why doesn't this work? PETSc knows where HDF5 is, no matter how it is specified, and you can query it from Python.
How do I do this? I thought it was only possible by parsing the output of "make info". Thanks, Lawrence
On Fri, Aug 23, 2019 at 12:16 PM Lawrence Mitchell <wence@gmx.li> wrote:
On 23 Aug 2019, at 17:03, Matthew Knepley <knepley@gmail.com> wrote:
[...]
Our issue is that we basically rely on PETSc to build most of the compiled dependencies, and hence if PETSc was built with --with-hdf5-dir=... (rather than --download-hdf5) we don't find it.
Why doesn't this work? PETSc knows where HDF5 is, no matter how it is specified, and you can query it from Python.
How do I do this? I thought it was only possible by parsing the output of "make info".
Here is an example: https://gitlab.com/petsc/petsc/blob/master/lib/petsc/bin/petsc_conf.py You would want to pull out the HDF5 module. Thanks, Matt
Thanks,
Lawrence
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
When you compiled PETSc by hand, which configure options did you use? We depend on PETSc having downloaded quite a few things for us, in particular including hdf5 . It looks like you didn’t build PETSc with that option. If you run: firedrake-install --show-petsc-options the installer will list the options it would have passed to PETSc’s configure. You should ensure that you at least pass those options. Regards, David From: "Madden, Niall" <niall.madden@nuigalway.ie> Date: Thursday, 22 August 2019 at 22:57 To: "Ham, David A" <david.ham@imperial.ac.uk>, Lawrence Mitchell <wence@gmx.li> Cc: "Sagiyama, Koki" <k.sagiyama@imperial.ac.uk>, firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi guys, Thanks for the replies. Lawrence:
Given you started with a clean conda environment, did you run into issues just running firedrake-install inside that? Yes. Fails when compiling Petsc. Since I can compile Petsc by hand, I tried that passed --honour-petsc-dir when trying to build firedrake. But then it fails to compile h5py The log reports "/MYOB/firedrake/src/h5py/h5py/api_compat.h(27): catastrophic error: cannot open source file "hdf5.h""
David:
One more piece of the puzzle for you: the activate script is part of Python venv. One of the first things that the install script does is create a Firedrake venv into which Firedrake is installed. It is the creation of the venv which creates the activate script. I see. Thanks.
I would reiterate what Lawrence says: installing by hand like this is very, very error-prone. So too is not installing by hand it seems. At least for my setup.
In particular, installing by hand and then hoping that the update script will work sounds very optimistic indeed. I share your pessimism. I hadn't expressed that hope, I just noted that that file hadn't been created.
I guess I could try building Python, as David suggested earlier. (Or going back to FEniCS, which installed without problems - but I do like quads :-) Niall. ________________________________ From: Ham, David A <david.ham@imperial.ac.uk> Sent: Thursday 22 August 2019 08:48 To: Lawrence Mitchell <wence@gmx.li>; Madden, Niall <niall.madden@nuigalway.ie> Cc: Sagiyama, Koki <k.sagiyama@imperial.ac.uk>; firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, One more piece of the puzzle for you: the activate script is part of Python venv. One of the first things that the install script does is create a Firedrake venv into which Firedrake is installed. It is the creation of the venv which creates the activate script. I would reiterate what Lawrence says: installing by hand like this is very, very error-prone. It would almost certainly be better to run the install script. In particular, installing by hand and then hoping that the update script will work sounds very optimistic indeed. Regards, David On 21/08/2019, 16:22, "Lawrence Mitchell" <wence@gmx.li> wrote: Hi Niall, On Wed, 21 Aug 2019 at 15:04, Madden, Niall <niall.madden@nuigalway.ie> wrote: > > Hi all, > I've returned to this task, and tried to follow David's suggestion > of building on a clean conda environment. This time, I used the intel compiler (mpiicc). > > Trying to follow the steps in the install script, we cloned > firedrake, ufl, fiat, PyOP2, COFFEE, loopy, petsc, supermesh, spatialindex, etc. > > Then we can successfully install all these. In particular, we can > configure/make/make install petsc > > Finally, we run, in firedrake, > python setup.py install > with LDSHARED, CFLAGS, CC, CXX, and prefix set appropriately. [...] > That is, if I try > echo "import firedrake" | python > I get > > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "/MYOB/r/libs/firedrake/firedrake/__init__.py", line 5, in <module> > if "PETSC_DIR" in os.environ and not config["options"]["honour_petsc_dir"]: > TypeError: 'NoneType' object is not subscriptable [...] > I note that, the "activate" script is not in firedrake/bin > Neither, as it happens, is firedrake-update, but firedrake-clean, firedrake-install, and firedrake-zenodo are. This is an idiosyncrasy of the firedrake-install process. "firedrake-update" is actually just a copy of "firedrake-install" with a different name (so it knows it's in update mode). firedrake-{clean/install/zenodo} are created in the bin directory by setup.py, but firedrake-update is copied _by the firedrake-install process_. Hence it not existing because you did everything by hand. You can "fix" this by just copying firedrake-install to firedrake-update in the bin directory. The issue with the TypeError above is related. As well as creating the firedrake-update script, firedrake-install also writes a little configuration file that remembers the options you installed firedrake with (so that firedrake-update maintains them). Again, because you didn't run firedrake-install, this file doesn't exist. The relevant file to be created is `configuration.json` in the firedrake_configuration subdirectory of the firedrake source tree. Mine looks like: {"options": {"package_manager": false, "minimal_petsc": false, "mpicc": null, "mpicxx": null, "mpif90": null, "mpiexec": null, "disable_ssh": false, "honour_petsc_dir": true, "show_petsc_configure_options": false, "slepc": true, "packages": [], "honour_pythonpath": false, "petsc_int_type": "int32", "cache_dir": "/home/vtdb72/Documents/work/src/firedrake/.cache"}, "environment": {}, "additions": []} Editing this by hand is, however, rather error prone. Additionally, I'm not necessarily sure that firedrake-update will run as normal. Given you started with a clean conda environment, did you run into issues just running firedrake-install inside that? Thanks, Lawrence
On 23 Aug 2019, at 11:41, Ham, David A <david.ham@imperial.ac.uk> wrote:
When you compiled PETSc by hand, which configure options did you use? We depend on PETSc having downloaded quite a few things for us, in particular including hdf5 . It looks like you didn’t build PETSc with that option.
If you run:
firedrake-install --show-petsc-options
--show-petsc-configure-options ^^^^^^^^^
the installer will list the options it would have passed to PETSc’s configure. You should ensure that you at least pass those options.
Thanks, Lawrence
Hello again, Its Friday afternoon, so I'm back poking at the Firedrake/HPC/no apt problem. First, thanks for all the helpful replies.
building Python from source is far easier than manually installing Firedrake. True! I built it without any problems using gcc, and with only minor issues using icc. So, I am now Conda free - thanks!
However, even with it firedrake-install fails when it tries to build Petsc. Specifically, it couldn't find the BLAS, and so needs Petsc to be installed with --download-fblaslapack=1 But I can't see how to pass that option to the firedrake installer. So, I'm back to installing Petsc by hand, which **seems** to work OK. No error is reported in by configure; make; make install. I can run all the demos in /SOMWHERE/petsc/arch-linux-c-debug/externalpackages/git.petsc4py/demo So that's good (right?). Next...
I think this is actually easy to override: we only need to interpose in one place. Can you try using the firedrake-install script from here: https://raw.githubusercontent.com/firedrakeproject/firedrake/wence/fix/insta... You can then do: export HDF5_DIR=/path/you/used/for/--with-hdf5-dir python3 firedrake-install --honour-petsc-dir Thanks for this. It certainly helps. Although installation still fails, at least the log file (attached) is twice as long. Up to 48k lines from my previous best of 21k lines. Progress!
It seems to break when installing PyOP. Specifically, /MYOB/with-my-python/firedrake/bin/python -m pip install --no-binary mpi4py,randomgen -vvv -r PyOP2/requirements-ext.txt The issue here seems to be with mpi4py. Frustratingly, python -m pip install --no-binary mpi4py,randomgen -vvv -r PyOP2/requirements-ext.txt succeeds. Thoughts? FYI, python3 firedrake-install -v --disable-ssh --no-package-manager --mpicc mpiicc --mpicxx mpiicpc --mpif90 mpiifort --mpiexec mpiexec --honour-pythonpath --honour-petsc-dir --show-petsc-configure-options returns with ********************************************* Would build PETSc with the following options: ********************************************* --download-chaco --download-eigen=/ichec/home/users/niall/with-my-python/firedrake/src/eigen-3.3.3.tgz --download-exodusii --download-hdf5 --download-hypre --download-metis --download-ml --download-mumps --download-netcdf --download-pastix --download-pnetcdf --download-pragmatic --download-ptscotch --download-scalapack --download-suitesparse --download-superlu_dist --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-zlib Eigen will be downloaded from https://bitbucket.org/eigen/eigen/get/3.3.3.tar.gz ---end---
apologies that this is still not working, although we would like to get to the bottom. No apologies needed! I can still do dev work on my Ubuntu laptop, where firedrake installed out-of-the-box. Getting it running in the cluster will be great, eventually, but not urgent.
Have a good weekend. Niall. Dr Niall Madden, School of Mathematics, Statistics and Applied Mathematics, NUI Galway Vice-dean for Equality, Diversity and Inclusion, College of Science Web: http://www.maths.nuigalway.ie/~niall Local co-organiser: 23rd Conference of the International Linear Algebra Society - www.ilas2020.ie<http://ilas2020.ie/> Associate Editor of Numerical Algorithms (Springer) ________________________________ From: Ham, David A <david.ham@imperial.ac.uk> Sent: Friday 23 August 2019 11:41 To: Madden, Niall <niall.madden@nuigalway.ie>; Lawrence Mitchell <wence@gmx.li> Cc: Sagiyama, Koki <k.sagiyama@imperial.ac.uk>; firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager When you compiled PETSc by hand, which configure options did you use? We depend on PETSc having downloaded quite a few things for us, in particular including hdf5 . It looks like you didn’t build PETSc with that option. If you run: firedrake-install --show-petsc-options the installer will list the options it would have passed to PETSc’s configure. You should ensure that you at least pass those options. Regards, David From: "Madden, Niall" <niall.madden@nuigalway.ie> Date: Thursday, 22 August 2019 at 22:57 To: "Ham, David A" <david.ham@imperial.ac.uk>, Lawrence Mitchell <wence@gmx.li> Cc: "Sagiyama, Koki" <k.sagiyama@imperial.ac.uk>, firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi guys, Thanks for the replies. Lawrence:
Given you started with a clean conda environment, did you run into issues just running firedrake-install inside that?
Yes. Fails when compiling Petsc. Since I can compile Petsc by hand, I tried that passed --honour-petsc-dir when trying to build firedrake. But then it fails to compile h5py The log reports "/MYOB/firedrake/src/h5py/h5py/api_compat.h(27): catastrophic error: cannot open source file "hdf5.h"" David:
One more piece of the puzzle for you: the activate script is part of Python venv. One of the first things that the install script does is
create a Firedrake venv into which Firedrake is installed. It is the creation of the venv which creates the activate script.
I see. Thanks.
I would reiterate what Lawrence says: installing by hand like this is very, very error-prone.
So too is not installing by hand it seems. At least for my setup.
In particular, installing by hand and then hoping that the update script will work sounds very optimistic indeed.
I share your pessimism. I hadn't expressed that hope, I just noted that that file hadn't been created. I guess I could try building Python, as David suggested earlier. (Or going back to FEniCS, which installed without problems - but I do like quads :-) Niall. ________________________________ From: Ham, David A <david.ham@imperial.ac.uk> Sent: Thursday 22 August 2019 08:48 To: Lawrence Mitchell <wence@gmx.li>; Madden, Niall <niall.madden@nuigalway.ie> Cc: Sagiyama, Koki <k.sagiyama@imperial.ac.uk>; firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager Hi Niall, One more piece of the puzzle for you: the activate script is part of Python venv. One of the first things that the install script does is create a Firedrake venv into which Firedrake is installed. It is the creation of the venv which creates the activate script. I would reiterate what Lawrence says: installing by hand like this is very, very error-prone. It would almost certainly be better to run the install script. In particular, installing by hand and then hoping that the update script will work sounds very optimistic indeed. Regards, David On 21/08/2019, 16:22, "Lawrence Mitchell" <wence@gmx.li> wrote: Hi Niall, On Wed, 21 Aug 2019 at 15:04, Madden, Niall <niall.madden@nuigalway.ie> wrote: > > Hi all, > I've returned to this task, and tried to follow David's suggestion > of building on a clean conda environment. This time, I used the intel compiler (mpiicc). > > Trying to follow the steps in the install script, we cloned > firedrake, ufl, fiat, PyOP2, COFFEE, loopy, petsc, supermesh, spatialindex, etc. > > Then we can successfully install all these. In particular, we can > configure/make/make install petsc > > Finally, we run, in firedrake, > python setup.py install > with LDSHARED, CFLAGS, CC, CXX, and prefix set appropriately. [...] > That is, if I try > echo "import firedrake" | python > I get > > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "/MYOB/r/libs/firedrake/firedrake/__init__.py", line 5, in <module> > if "PETSC_DIR" in os.environ and not config["options"]["honour_petsc_dir"]: > TypeError: 'NoneType' object is not subscriptable [...] > I note that, the "activate" script is not in firedrake/bin > Neither, as it happens, is firedrake-update, but firedrake-clean, firedrake-install, and firedrake-zenodo are. This is an idiosyncrasy of the firedrake-install process. "firedrake-update" is actually just a copy of "firedrake-install" with a different name (so it knows it's in update mode). firedrake-{clean/install/zenodo} are created in the bin directory by setup.py, but firedrake-update is copied _by the firedrake-install process_. Hence it not existing because you did everything by hand. You can "fix" this by just copying firedrake-install to firedrake-update in the bin directory. The issue with the TypeError above is related. As well as creating the firedrake-update script, firedrake-install also writes a little configuration file that remembers the options you installed firedrake with (so that firedrake-update maintains them). Again, because you didn't run firedrake-install, this file doesn't exist. The relevant file to be created is `configuration.json` in the firedrake_configuration subdirectory of the firedrake source tree. Mine looks like: {"options": {"package_manager": false, "minimal_petsc": false, "mpicc": null, "mpicxx": null, "mpif90": null, "mpiexec": null, "disable_ssh": false, "honour_petsc_dir": true, "show_petsc_configure_options": false, "slepc": true, "packages": [], "honour_pythonpath": false, "petsc_int_type": "int32", "cache_dir": "/home/vtdb72/Documents/work/src/firedrake/.cache"}, "environment": {}, "additions": []} Editing this by hand is, however, rather error prone. Additionally, I'm not necessarily sure that firedrake-update will run as normal. Given you started with a clean conda environment, did you run into issues just running firedrake-install inside that? Thanks, Lawrence
participants (6)
-
Ham, David A
-
Lawrence Mitchell
-
Madden, Niall
-
Matthew Knepley
-
Miklós Homolya
-
Sagiyama, Koki