Hello, Has anybody had a chance to look at this? Being able to use multi-node Firedrake would greatly accelerate my work, and I would be grateful for any suggestions. Aaron ________________________________ From: Aaron Matthew Baier-Reinio Sent: July 15, 2018 9:24:33 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello again, It seems that warming the cache fixes the problem when the mesh resolution is small, but when the mesh resolution is increased the problem still occurs. As a MWE of this I have a file test.py that contains the following code: from firedrake import * from mpi4py import MPI set_log_level(logging.INFO) size = MPI.COMM_WORLD.Get_size() rank = MPI.COMM_WORLD.Get_rank() name = MPI.Get_processor_name() nn = 10 mesh = BoxMesh(nn, nn, nn, 1.0, 1.0, 1.0) DG = FunctionSpace(mesh, "DG", 1) expr = Expression("1.0", degree=1) f = project(expr, DG) print("Finished on process %d of %d on %s." % (rank, size, name)) To test this I run a Slurm job on 2 nodes with 1 CPU per node, so that "mpirun -np 1 python3 test.py" will run the script on one node only, whereas "-np 2" will run the script on two nodes. Now, with nn=10, if I run the following in sequence, mpirun -np 2 python3 test.py mpirun -np 1 python3 test.py mpirun -np 2 python3 test.py the first mpirun will fail as expected, but the last two both succeed. However, doing the same thing but with nn=40, both "-np 2" runs will fail. I have attached the output for both cases. Is what I am doing to warm the cache correct or is there a different way I should be doing it that will work for larger meshes? Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> Sent: July 14, 2018 2:17 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello Miklos and Tuomas, Thank you for the replies. I am not sure what is going on with the MPI fork warning. This warning message is always shown as soon as I import firedrake. It will show up no matter how many cores or nodes I am running on, and even if I run the script without using mpirun, it still shows up. I have just been ignoring it. Regarding your advice about warming the cache and/or changing the cache directories, I will try doing both of these things. I will let you know what happens. Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Tuomas Karna <tuomas.karna@gmail.com> Sent: July 14, 2018 1:04:39 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hi Miklos and Aaron, I agree that this looks like a compilation/cache issue. For what it's worth, however, on some linux clusters I do see the MPI fork warning but it does not affect firedrake; compilation works just fine in parallel. It might be worth trying to point the cache directories to a different location. I think they are set with these env variables: FIREDRAKE_TSFC_KERNEL_CACHE_DIR PYOP2_CACHE_DIR Cheers, Tuomas On 07/14/2018 01:48 PM, Homolya, Miklós wrote: Hello, The relevant message seems to be this: A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. Firedrake compiles the high-level specification of your PDE into efficient binary code dynamically, just in time. It first generates C code, that calls the host C compiler to compile that. This step fails because your MPI does not support the fork() system call. Suggested workaround is to warm up your disk cache by first solving a small problem serially (or on a single node if that works), and then launch your parallel runs with warm cache, so Firedrake won't need to do further compilations. Regards, Miklos ________________________________ From: firedrake-bounces@imperial.ac.uk<mailto:firedrake-bounces@imperial.ac.uk> <firedrake-bounces@imperial.ac.uk><mailto:firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca><mailto:ambaierreinio@edu.uwaterloo.ca> Sent: 13 July 2018 19:15:51 To: firedrake Subject: [firedrake] Errors running Firedrake on multiple nodes Hello Firedrake team, I am trying to install Firedrake on a cluster and use mpi to run my Firedrake scripts on multiple nodes. I am running into some issues and was wondering if you would be able to help. The operating system on the cluster is CentOS 7.4. Both Python 3.5 and PETSc 3.9.0 are already loaded on the cluster via the environment modules tool. So, to install I am running the install script as follows: python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir --honour-pythonpath The install seems to go through successfully. The resultant installation seems to work when running on a single node, but I am getting errors when I run on multiple nodes. If I run my script as mpirun -np 8 python3 script.py and all 8 CPUs are located on the same node, then the script seems to run successfully. However, if the CPUs are distributed across multiple nodes, then I always get error messages. The general pattern to these error messages is that I will first get a PyOP2 caching error that looks like this: Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 20, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('WRITE'), <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 4, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('READ'), <class 'pyop2.sequential.Arg'>, (1,), dtype('float64'), Access('READ')) This error will then be followed by a bunch of "During handling of the above exception, another exception occurred" errors. I have attached the full output from one of these error messages. Do you have any idea what could be going wrong here? Please let me know if you need more information. Thanks, Aaron _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk<mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake