-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/08/15 18:16, Justin Chang wrote:
Hi Lawrence,
After several trial and run experimentations these last few days, I still get the warning even with the pre-fork branches. However, if I use our system's intel-compiled libraries and compilers, I get no such errors when using mpirun. I suspect that this may have something to do with the HPC system that I am running on.
Reading between the lines, it seems to be some bad interaction between OpenMPI, forking, and the infiniband OpenFabrics transport. I have filed a bug that will remind us about it: I think our local HPC system also has the same setup, but I have not run there recently.
But I am now running into a stranger issue:
I have attached an RT0 code that I am working with. Basically, it takes as input the seed size (aka the number of cells to generate in each spatial direction). The command line argument goes like this: "mpirun -n 1 python Test_RT0.py <seed>"
I get strange errors when the seed number changes. I ran all problems with -log_trace. For <seed> = 3, i get this:
$ python Test_RT0.py 3
...
pyop2.exceptions.CompilationError: Command "['mpicc', '-std=c99', '-fPIC', '-Wall', '-g', '-O3', '-fno-tree-vectorize', '-I/home/jchang23/petsc-dev/include', '-I/home/jchang23/petsc-dev/arch-linux2-c-opt/include', '-I/home/jchang23/firedrake-deps/firedrake/firedrake', '-I/home/jchang23/firedrake-deps/PyOP2/pyop2', '-msse', '-o', '/tmp/pyop2-cache-uid3003/32e0bd01cb649f218f2092c503c1d41f.so.tmp',
'/tmp/pyop2-cache-uid3003/32e0bd01cb649f218f2092c503c1d41f.c',
'-shared', '-L/home/jchang23/petsc-dev/lib', '-L/home/jchang23/petsc-dev/arch-linux2-c-opt/lib', '-Wl,-rpath,/home/jchang23/petsc-dev/lib', '-Wl,-rpath,/home/jchang23/petsc-dev/arch-linux2-c-opt/lib', '-lpetsc', '-lm']" returned with error.
Unable to compile code
This error, I think, comes from using intel compilers, but passing gcc options. Can you try with: parameters["pyop2_options"]["compiler"] = "intel" which has some prebaked options for the intel compiler.
For <seed>=5 I get this:
petsc4py.PETSc.Error: error code 77
[0] DMPlexGenerate() line 1080 in /home/jchang23/petsc-dev/src/dm/impls/plex/plexgenerate.c
[0] DMPlexGenerate_CTetgen() line 834 in /home/jchang23/petsc-dev/src/dm/impls/plex/plexgenerate.c
[0] TetGenTetrahedralize() line 21483 in /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
[0] TetGenMeshDelaunizeVertices() line 12113 in /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
[0] TetGenMeshDelaunayIncrFlip() line 12046 in /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
[0] TetGenMeshInsertVertexBW() line 11559 in /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
[0] TetGenMeshInSphereS() line 5411 in /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
[0] Petsc has generated inconsistent data
[0] This is wrong
I am very confused by these strange results. Any explanation for these?
This one, I /think/ I've seen before. I believe that the intel compiler produces bad code for ctetgen with optimisations on. I think this is plausibly a petsc-only problem, try with the same setup but do: from petsc4py import PETSc bdy = PETSc.DMPlex().create() bdy.setDimension(2) bdy.createCubeBoundary([0, 0, 0], [1, 1, 1], [5, 5, 5]) dm = PETSc.DMPlex().generate(bdy) I don't have access to the intel compiler stack here, but maybe it's worth reporting the bug in this form (assuming it shows up) to petsc-maint. If you're able to use gmsh to build the appropriate meshes, that is probably a more robust option. Alternately you could try (although I never have) using tetgen (rather than ctetgen). I think you'll need to configure PETSc with --with-clanguage=c++ --download-tetgen (rather than --download-ctetgen). Cheers, Lawrence -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJVybRbAAoJECOc1kQ8PEYvvS0IALzJjuMdyjvdn5H4RQtNAuHy UiMVhPWqE6F9iIeTVBY+Or5YGjlTKFHBXKjZCl5Fs3TgK8GDsn0COuoy+9TWzMR9 Kn/XnxZuN1NYGwuiPlT8plahMGc2jAkI7peiPJsQyAfOfVA6nlBi6mDLiEDHuuvX 7g/9P/ZcWu89D9/9m/mfNTRZq9anF5zcXjIHHJzojdlg8DBWviAFO7gVpqivrD72 Zzr+Q3xaFKQI9deNkjFacjJPJT/OUkBKAf/n4BD5wNAPt6z/TCyTJ4fTdeZ4lY2I 2m6IDyJE4RNlmaPfzqq8D91T/kswy0wbC3IrMwTOGG47GoEYuOXqe45llZ/+M4U= =HGNY -----END PGP SIGNATURE-----