On 04/10/14 13:15, Eike Mueller wrote:
Dear firedrakers,
I tried to run the Helmholtz multigrid solver on higher core counts on ARCHER. In certain configurations it works, but in others it crashes.
The number of fine grid cells is 20*4^(4+refcount_coarse).
These mesh sizes should all be manageable afaict, I have run on larger meshes (up to ~60M cells afair).
In particular, I was not able to run on 48 cores (i.e. 2 full nodes)
1 node, 16 cores/node = 16 cores, refcount_coarse=5: Runs 2 nores, 16 cores/node = 48 cores, refcount_coarse=4: hangs without printing error message
I presume you mean 24 cores/node? How long did this run for? Was it eventually killed by the scheduler?
2 nores, 16 cores/node = 48 cores, refcount_coarse=5: crashes with PETSc error message
This isn't a PETSc error, it's only PETSc reporting the crash. The actual issue is a compile error. Can you check whether /work/n02/n02/eike//pyop2-cache/9b5f79130a429e4044a23dacb78191df.so or .so.tmp exists?
2 nores, 16 cores/node = 48 cores, refcount_coarse=5: crashes with different error message
This is actually the same error. Does /work/n02/n02/eike//pyop2-cache/c8aac6eec48490d587593472c04a5557.so or .so.tmp exist?
Output logs for all cases are attached, as well as the pbs scripts for running on 48 cores.
The job script appears to be for a 32 core run. It looks legit, though you wouldn't need to load the firedrake module given that you also load the dependency modules and use your own PyOP2/Firedrake
I also tried running the poisson example in firedrake-benchmark in different configurations:
1 node, 16 cores/node = 16 cores => works 2 nodes, 8 cores/node = 16 cores => works 2 nodes, 24 cores/node = 48 cores => crashes with PETSc error
This one is baffling: it appears that only ranks 24-47 are aborted. I have absolutely no clue what is going on. Is this reproducible?
I did use the generic ARCHER firedrake installation in this case, though. In the Helmholtz solver runs I used my own version of firedrake and PyOP2 since I need the multigrid and local_par-loop branches.
This makes me suspicious that it is not a problem specific to my code, but then I thought that you did run firedrake on relatively large core counts before, didn't you?
I have run on up to 1536 cores for Poisson and Cahn-Hilliard.
The output of the Poisson benchmarks is also attached.
Any ideas what might be going wrong and what is the best way for debugging this?
I'm not entirely sure what to suggest about the run that just hangs other than trying to pinpoint where exactly that happens. For the compilation failures, I recall I had those sporadically and could never figure out what exactly had gone wrong. Check what is in the cache and also what the compiled code represents: and assembly or an expression? Florian
Thanks,
Eike