On 30/03/17 14:45, Matthew Knepley wrote:
On Thu, Mar 30, 2017 at 8:39 AM, Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk <mailto:lawrence.mitchell@imperial.ac.uk>> wrote:
On 30/03/17 14:28, Matthew Knepley wrote: > What is going on here? It looks like you are using subprocess. Why > would you do that on a cluster rather than MPI?
It's all pretty icky. One of the JIT steps involves forking something. Plausibly, Henrik, your MPI doesn't like fork. I went round at one point to try and set up a single subprocess that was able to fork before MPI_Init.
For example, when using craypich, I need to say:
export MPICH_GNI_FORK_MODE=FULLCOPY
Old OpenMPI versions, especially those using some infiniband stuff, don't like fork either, for example https://www.open-mpi.org/faq/?category=openfabrics#ofa-fork <https://www.open-mpi.org/faq/?category=openfabrics#ofa-fork>
Matt, here is David's response (you're not subscribed I think, so didn't get it):
Is there a simple thing we can do to shut this off for testing? He is on a cluster I believe.
I think, do: from firedrake import * parameters["coffee"]["optlevel"] = "O0" parameters["pyop2_options"]["opt_level"] = "O0" YOUR CODE HERE (Yes, this is terrible, WIP to create only one place you have to set this) This will turn off the coffee transformations. In the general case, you don't want to do this, but if you're attempting to weak scale a fieldsplit solve, it probably won't have a big effect. Lawrence