firedrake broken on ARCHER after update
Dear all, I’m trying to update my firedrake on ARCHER. I updated all relevant packages (fiat, petsc, coffee, ufl, ffc, PyOP2 and firedrake) but when I try to run firedrake-clean I get the error below. Any idea what this could be? Thanks, Eike /opt/modules/default/init/bash: line 11: unalias: pip: not found Traceback (most recent call last): File "/var/opt/cray/alps/spool/13759147/firedrake-clean", line 3, in <module> from firedrake.ffc_interface import clear_cache, FFCKernel File "/fs2/n02/n02/eike/git_workspace/firedrake/firedrake/__init__.py", line 2, in <module> import petsc File "/fs2/n02/n02/eike/git_workspace/firedrake/firedrake/petsc.py", line 4, in <module> petsc4py.init(sys.argv) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/__init__.py", line 42, in init PETSc = petsc4py.lib.ImportPETSc(arch) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 29, in ImportPETSc return Import('petsc4py', 'PETSc', path, arch) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 64, in Import module = imp.load_module(fullname, fo, fn, stuff) ImportError: /work/n02/n02/eike/git_workspace/petsc/cray-gnu-shared/lib/libpetsc.so.3.05: undefined symbol: PetscDLLibraryRegister_petscts -- Dr Eike Hermann Mueller Lecturer in Scientific Computing Department of Mathematical Sciences University of Bath Bath BA2 7AY, United Kingdom +44 1225 38 5557 e.mueller@bath.ac.uk<mailto:e.mueller@bath.ac.uk> http://people.bath.ac.uk/em459/
On 24 Apr 2015, at 18:26, Eike Mueller <E.Mueller@bath.ac.uk> wrote:
Dear all,
I’m trying to update my firedrake on ARCHER. I updated all relevant packages (fiat, petsc, coffee, ufl, ffc, PyOP2 and firedrake) but when I try to run firedrake-clean I get the error below.
Any idea what this could be?
Thanks,
Eike
/opt/modules/default/init/bash: line 11: unalias: pip: not found Traceback (most recent call last): File "/var/opt/cray/alps/spool/13759147/firedrake-clean", line 3, in <module> from firedrake.ffc_interface import clear_cache, FFCKernel File "/fs2/n02/n02/eike/git_workspace/firedrake/firedrake/__init__.py", line 2, in <module> import petsc File "/fs2/n02/n02/eike/git_workspace/firedrake/firedrake/petsc.py", line 4, in <module> petsc4py.init(sys.argv) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/__init__.py", line 42, in init PETSc = petsc4py.lib.ImportPETSc(arch) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 29, in ImportPETSc return Import('petsc4py', 'PETSc', path, arch) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 64, in Import module = imp.load_module(fullname, fo, fn, stuff) ImportError: /work/n02/n02/eike/git_workspace/petsc/cray-gnu-shared/lib/libpetsc.so.3.05: undefined symbol: PetscDLLibraryRegister_petscts
That looks somehow like the petsc install didn't work, said symbol ought to exist in the petsc library. Does a simple petsc program compile and run? For example, in the petsc source tree: cd src/snes/examples/tutorials make ex19 aprun -n 1 -N 1 ./ex19 Lawrence
Thanks, I will try that. It's odd, since I reconfigured and did a make clean; make in the PETSc directory, and I checked that the generated library has an up-to-date modification date. Maybe I should clean out the entire PETSc directory and reclone it. Eike PS: i did indeed update petsc4py as well and only forgot in in the list, sorry about the confusion... Sent from my iPad
On 24 Apr 2015, at 18:50, Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk> wrote:
On 24 Apr 2015, at 18:26, Eike Mueller <E.Mueller@bath.ac.uk> wrote:
Dear all,
I’m trying to update my firedrake on ARCHER. I updated all relevant packages (fiat, petsc, coffee, ufl, ffc, PyOP2 and firedrake) but when I try to run firedrake-clean I get the error below.
Any idea what this could be?
Thanks,
Eike
/opt/modules/default/init/bash: line 11: unalias: pip: not found Traceback (most recent call last): File "/var/opt/cray/alps/spool/13759147/firedrake-clean", line 3, in <module> from firedrake.ffc_interface import clear_cache, FFCKernel File "/fs2/n02/n02/eike/git_workspace/firedrake/firedrake/__init__.py", line 2, in <module> import petsc File "/fs2/n02/n02/eike/git_workspace/firedrake/firedrake/petsc.py", line 4, in <module> petsc4py.init(sys.argv) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/__init__.py", line 42, in init PETSc = petsc4py.lib.ImportPETSc(arch) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 29, in ImportPETSc return Import('petsc4py', 'PETSc', path, arch) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 64, in Import module = imp.load_module(fullname, fo, fn, stuff) ImportError: /work/n02/n02/eike/git_workspace/petsc/cray-gnu-shared/lib/libpetsc.so.3.05: undefined symbol: PetscDLLibraryRegister_petscts
That looks somehow like the petsc install didn't work, said symbol ought to exist in the petsc library. Does a simple petsc program compile and run? For example, in the petsc source tree:
cd src/snes/examples/tutorials make ex19 aprun -n 1 -N 1 ./ex19
Lawrence
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Hi, it’s working again on ARCHER now, I cleared out my entire PETSc directory and cloned and rebuilt it again. My code runs fine on 24 cores, although on ARCHER I also still see the problem with the PETSc field_split solver, which takes much, much longer than previously. Running on 24 cores (1 full node) and comparing the results of old PETSc runs I get on a full node at lowest order: 13.80 (PETSc field_split) vs. 11.23 (matrixfree). At higher order, I don’t see to be able to run the matrix-free solver in the original configuration any more: it works on one core (with a reduced problem size), but not on a full node. I did check on my laptop that I can still run in parallel, but the results of a 1 core run and a 4 core run are not exactly identical at higher order, so something is not quite right there. Thanks, Eike
On 24 Apr 2015, at 22:48, Eike Mueller <E.Mueller@bath.ac.uk> wrote:
Thanks, I will try that. It's odd, since I reconfigured and did a
make clean; make
in the PETSc directory, and I checked that the generated library has an up-to-date modification date. Maybe I should clean out the entire PETSc directory and reclone it.
Eike
PS: i did indeed update petsc4py as well and only forgot in in the list, sorry about the confusion...
Sent from my iPad
On 24 Apr 2015, at 18:50, Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk> wrote:
On 24 Apr 2015, at 18:26, Eike Mueller <E.Mueller@bath.ac.uk> wrote:
Dear all,
I’m trying to update my firedrake on ARCHER. I updated all relevant packages (fiat, petsc, coffee, ufl, ffc, PyOP2 and firedrake) but when I try to run firedrake-clean I get the error below.
Any idea what this could be?
Thanks,
Eike
/opt/modules/default/init/bash: line 11: unalias: pip: not found Traceback (most recent call last): File "/var/opt/cray/alps/spool/13759147/firedrake-clean", line 3, in <module> from firedrake.ffc_interface import clear_cache, FFCKernel File "/fs2/n02/n02/eike/git_workspace/firedrake/firedrake/__init__.py", line 2, in <module> import petsc File "/fs2/n02/n02/eike/git_workspace/firedrake/firedrake/petsc.py", line 4, in <module> petsc4py.init(sys.argv) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/__init__.py", line 42, in init PETSc = petsc4py.lib.ImportPETSc(arch) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 29, in ImportPETSc return Import('petsc4py', 'PETSc', path, arch) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 64, in Import module = imp.load_module(fullname, fo, fn, stuff) ImportError: /work/n02/n02/eike/git_workspace/petsc/cray-gnu-shared/lib/libpetsc.so.3.05: undefined symbol: PetscDLLibraryRegister_petscts
That looks somehow like the petsc install didn't work, said symbol ought to exist in the petsc library. Does a simple petsc program compile and run? For example, in the petsc source tree:
cd src/snes/examples/tutorials make ex19 aprun -n 1 -N 1 ./ex19
Lawrence
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Hi,
At higher order, I don’t see to be able to run the matrix-free solver in the original configuration any more: it works on one core (with a reduced problem size), but not on a full node. I did check on my laptop that I can still run in parallel, but the results of a 1 core run and a 4 core run are not exactly identical at higher order, so something is not quite right there.
Hang on, thinking about this again, if I use BJACOBI+ILU as the preconditioner for the velocity mass inverse, isn’t the result naturally dependent on the number of processors, since by default a block corresponds to the dofs stored on one process? At lowest order I use the inverse of the diagonal lumped mass matrix, so there I don’t see this. However, if I change BJACOBI+ILU -> JACOBI I still get a difference between the results on 1 and on 4 processors, so I must be missing something else. Thanks, Eike
On 24 Apr 2015, at 22:48, Eike Mueller <E.Mueller@bath.ac.uk> wrote:
Thanks, I will try that. It's odd, since I reconfigured and did a
make clean; make
in the PETSc directory, and I checked that the generated library has an up-to-date modification date. Maybe I should clean out the entire PETSc directory and reclone it.
Eike
PS: i did indeed update petsc4py as well and only forgot in in the list, sorry about the confusion...
Sent from my iPad
On 24 Apr 2015, at 18:50, Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk> wrote:
On 24 Apr 2015, at 18:26, Eike Mueller <E.Mueller@bath.ac.uk> wrote:
Dear all,
I’m trying to update my firedrake on ARCHER. I updated all relevant packages (fiat, petsc, coffee, ufl, ffc, PyOP2 and firedrake) but when I try to run firedrake-clean I get the error below.
Any idea what this could be?
Thanks,
Eike
/opt/modules/default/init/bash: line 11: unalias: pip: not found Traceback (most recent call last): File "/var/opt/cray/alps/spool/13759147/firedrake-clean", line 3, in <module> from firedrake.ffc_interface import clear_cache, FFCKernel File "/fs2/n02/n02/eike/git_workspace/firedrake/firedrake/__init__.py", line 2, in <module> import petsc File "/fs2/n02/n02/eike/git_workspace/firedrake/firedrake/petsc.py", line 4, in <module> petsc4py.init(sys.argv) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/__init__.py", line 42, in init PETSc = petsc4py.lib.ImportPETSc(arch) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 29, in ImportPETSc return Import('petsc4py', 'PETSc', path, arch) File "/work/n02/n02/eike/git_workspace/petsc4py/cray-gnu-shared/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 64, in Import module = imp.load_module(fullname, fo, fn, stuff) ImportError: /work/n02/n02/eike/git_workspace/petsc/cray-gnu-shared/lib/libpetsc.so.3.05: undefined symbol: PetscDLLibraryRegister_petscts
That looks somehow like the petsc install didn't work, said symbol ought to exist in the petsc library. Does a simple petsc program compile and run? For example, in the petsc source tree:
cd src/snes/examples/tutorials make ex19 aprun -n 1 -N 1 ./ex19
Lawrence
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 26/04/15 13:53, Eike Mueller wrote:
Hi,
At higher order, I don’t see to be able to run the matrix-free solver in the original configuration any more: it works on one core (with a reduced problem size), but not on a full node. I did check on my laptop that I can still run in parallel, but the results of a 1 core run and a 4 core run are not exactly identical at higher order, so something is not quite right there. Hang on, thinking about this again, if I use BJACOBI+ILU as the preconditioner for the velocity mass inverse, isn’t the result naturally dependent on the number of processors, since by default a block corresponds to the dofs stored on one process? At lowest order I use the inverse of the diagonal lumped mass matrix, so there I don’t see this.
Yes, that's right, bjacobi+ilu first decouples the processes and then does ILU on the local block, so it will be different on different processes.
However, if I change BJACOBI+ILU -> JACOBI I still get a difference between the results on 1 and on 4 processors, so I must be missing something else.
In this situation I would recommend checking that the operator is indeed the same on 1 and 4 processors. Computing some of the leading eigenvalues is probably good enough, you can do this with SLEPc: pip install git+https://wence@bitbucket.org/slepc/slepc.git#egg=slepc pip install git+https://wence@bitbucket.org/slepc/slepc4py.git#egg=slepc4py A = assemble(form) petsc_mat = A.M.handle from slepc4py import SLEPc eps = SLEPc.EPS().create() eps.setOperators(petsc_mat) eps.setWhichEigenpairs(eps.Which.LARGEST_MAGNITUDE) eps.solve() nev = eps.getConverged() for i in range(nev): evalue = eps.getEigenvalue(i) print evalue Cheers, Lawrence -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJVPf+JAAoJECOc1kQ8PEYv650H/3ZGkCy/6G/BivFhSMyKNaCz HS3P4KO/sOcNEJSGum1YSktWvWa5YkRWX9xACZWRisO3TWKgTpBiqhS4aGuOn2Xh QrO3Uhyoj5dx1S5o69qQiZppsMjdPViRnw4/O3YEAw7xUiKbbJqBmb30yP2fLGKe q2Ce/kwPG8DfVf0Xl6Z4muKHHquG38YwqDCSMc6s4QbIQwi8PQI+fAWcBwbZ2Wag vuB9TOIJUXmBMMpdlk8YPr903iu1Qd4koxbP8kN0kHsORdJJzxOsYfotEEk5Fdm4 de0UjSUSKFP1Kj45ZbZLbXFrJFIrFlSaWRFQmQEmpck2H8AMYDsF8TMviH4dE4w= =RMZi -----END PGP SIGNATURE-----
participants (2)
-
Eike Mueller
-
Lawrence Mitchell