Errors running Firedrake on multiple nodes
Hello Firedrake team, I am trying to install Firedrake on a cluster and use mpi to run my Firedrake scripts on multiple nodes. I am running into some issues and was wondering if you would be able to help. The operating system on the cluster is CentOS 7.4. Both Python 3.5 and PETSc 3.9.0 are already loaded on the cluster via the environment modules tool. So, to install I am running the install script as follows: python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir --honour-pythonpath The install seems to go through successfully. The resultant installation seems to work when running on a single node, but I am getting errors when I run on multiple nodes. If I run my script as mpirun -np 8 python3 script.py and all 8 CPUs are located on the same node, then the script seems to run successfully. However, if the CPUs are distributed across multiple nodes, then I always get error messages. The general pattern to these error messages is that I will first get a PyOP2 caching error that looks like this: Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 20, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('WRITE'), <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 4, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('READ'), <class 'pyop2.sequential.Arg'>, (1,), dtype('float64'), Access('READ')) This error will then be followed by a bunch of "During handling of the above exception, another exception occurred" errors. I have attached the full output from one of these error messages. Do you have any idea what could be going wrong here? Please let me know if you need more information. Thanks, Aaron
Hello, The relevant message seems to be this: A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. Firedrake compiles the high-level specification of your PDE into efficient binary code dynamically, just in time. It first generates C code, that calls the host C compiler to compile that. This step fails because your MPI does not support the fork() system call. Suggested workaround is to warm up your disk cache by first solving a small problem serially (or on a single node if that works), and then launch your parallel runs with warm cache, so Firedrake won't need to do further compilations. Regards, Miklos ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> Sent: 13 July 2018 19:15:51 To: firedrake Subject: [firedrake] Errors running Firedrake on multiple nodes Hello Firedrake team, I am trying to install Firedrake on a cluster and use mpi to run my Firedrake scripts on multiple nodes. I am running into some issues and was wondering if you would be able to help. The operating system on the cluster is CentOS 7.4. Both Python 3.5 and PETSc 3.9.0 are already loaded on the cluster via the environment modules tool. So, to install I am running the install script as follows: python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir --honour-pythonpath The install seems to go through successfully. The resultant installation seems to work when running on a single node, but I am getting errors when I run on multiple nodes. If I run my script as mpirun -np 8 python3 script.py and all 8 CPUs are located on the same node, then the script seems to run successfully. However, if the CPUs are distributed across multiple nodes, then I always get error messages. The general pattern to these error messages is that I will first get a PyOP2 caching error that looks like this: Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 20, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('WRITE'), <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 4, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('READ'), <class 'pyop2.sequential.Arg'>, (1,), dtype('float64'), Access('READ')) This error will then be followed by a bunch of "During handling of the above exception, another exception occurred" errors. I have attached the full output from one of these error messages. Do you have any idea what could be going wrong here? Please let me know if you need more information. Thanks, Aaron
Using the above suggested workaround may seem to fail for poorly formulated time-stepping methods. Depending on how time-dependent parameters are formulated, it is possible that form compilation is triggered in every time step. To check whether this happens, you may increase log level by inserting set_log_level(logging.INFO) after the Firedrake import statement. If you see repeated calls to the form compiler in each time step, that's a sign that your problem formulation should be improved. Furthermore, you said you use a system provided PETSc build. This most likely has nothing to do with your present problem, but could be a source of further problems, for two reasons: 1) PETSc version. Release version of PETSc are often too old for Firedrake, while the master version might be broken. Firedrake maintains a snapshot of PETSc master that is tested and known to work with Firedrake. 2) PETSc configuration. Firedrake assumes that certain features, provided by external packages, are available through PETSc. This may or may not be the case depending on how PETSc was configured. firedrake-install can tell you what switches it passes to PETSc configure. ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk> Sent: 14 July 2018 11:48:36 To: firedrake Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello, The relevant message seems to be this: A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. Firedrake compiles the high-level specification of your PDE into efficient binary code dynamically, just in time. It first generates C code, that calls the host C compiler to compile that. This step fails because your MPI does not support the fork() system call. Suggested workaround is to warm up your disk cache by first solving a small problem serially (or on a single node if that works), and then launch your parallel runs with warm cache, so Firedrake won't need to do further compilations. Regards, Miklos ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> Sent: 13 July 2018 19:15:51 To: firedrake Subject: [firedrake] Errors running Firedrake on multiple nodes Hello Firedrake team, I am trying to install Firedrake on a cluster and use mpi to run my Firedrake scripts on multiple nodes. I am running into some issues and was wondering if you would be able to help. The operating system on the cluster is CentOS 7.4. Both Python 3.5 and PETSc 3.9.0 are already loaded on the cluster via the environment modules tool. So, to install I am running the install script as follows: python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir --honour-pythonpath The install seems to go through successfully. The resultant installation seems to work when running on a single node, but I am getting errors when I run on multiple nodes. If I run my script as mpirun -np 8 python3 script.py and all 8 CPUs are located on the same node, then the script seems to run successfully. However, if the CPUs are distributed across multiple nodes, then I always get error messages. The general pattern to these error messages is that I will first get a PyOP2 caching error that looks like this: Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 20, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('WRITE'), <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 4, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('READ'), <class 'pyop2.sequential.Arg'>, (1,), dtype('float64'), Access('READ')) This error will then be followed by a bunch of "During handling of the above exception, another exception occurred" errors. I have attached the full output from one of these error messages. Do you have any idea what could be going wrong here? Please let me know if you need more information. Thanks, Aaron
Hi Miklos and Aaron, I agree that this looks like a compilation/cache issue. For what it's worth, however, on some linux clusters I do see the MPI fork warning but it does not affect firedrake; compilation works just fine in parallel. It might be worth trying to point the cache directories to a different location. I think they are set with these env variables: FIREDRAKE_TSFC_KERNEL_CACHE_DIR PYOP2_CACHE_DIR Cheers, Tuomas On 07/14/2018 01:48 PM, Homolya, Miklós wrote:
Hello,
The relevant message seems to be this:
A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged.
Firedrake compiles the high-level specification of your PDE into efficient binary code dynamically, just in time. It first generates C code, that calls the host C compiler to compile that. This step fails because your MPI does not support the fork() system call.
Suggested workaround is to warm up your disk cache by first solving a small problem serially (or on a single node if that works), and then launch your parallel runs with warm cache, so Firedrake won't need to do further compilations.
Regards,
Miklos
------------------------------------------------------------------------ *From:* firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> *Sent:* 13 July 2018 19:15:51 *To:* firedrake *Subject:* [firedrake] Errors running Firedrake on multiple nodes
Hello Firedrake team,
I am trying to install Firedrake on a cluster and use mpi to run my Firedrake scripts on multiple nodes. I am running into some issues and was wondering if you would be able to help.
The operating system on the cluster is CentOS 7.4. Both Python 3.5 and PETSc 3.9.0 are already loaded on the cluster via the environment modules tool. So, to install I am running the install script as follows:
python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir --honour-pythonpath
The install seems to go through successfully. The resultant installation seems to work when running on a single node, but I am getting errors when I run on multiple nodes. If I run my script as
mpirun -np 8 python3 script.py
and all 8 CPUs are located on the same node, then the script seems to run successfully. However, if the CPUs are distributed across multiple nodes, then I always get error messages. The general pattern to these error messages is that I will first get a PyOP2 caching error that looks like this:
Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 20, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('WRITE'), <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 4, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('READ'), <class 'pyop2.sequential.Arg'>, (1,), dtype('float64'), Access('READ'))
This error will then be followed by a bunch of "During handling of the above exception, another exception occurred" errors. I have attached the full output from one of these error messages.
Do you have any idea what could be going wrong here? Please let me know if you need more information.
Thanks,
Aaron
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Hello Miklos and Tuomas, Thank you for the replies. I am not sure what is going on with the MPI fork warning. This warning message is always shown as soon as I import firedrake. It will show up no matter how many cores or nodes I am running on, and even if I run the script without using mpirun, it still shows up. I have just been ignoring it. Regarding your advice about warming the cache and/or changing the cache directories, I will try doing both of these things. I will let you know what happens. Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Tuomas Karna <tuomas.karna@gmail.com> Sent: July 14, 2018 1:04:39 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hi Miklos and Aaron, I agree that this looks like a compilation/cache issue. For what it's worth, however, on some linux clusters I do see the MPI fork warning but it does not affect firedrake; compilation works just fine in parallel. It might be worth trying to point the cache directories to a different location. I think they are set with these env variables: FIREDRAKE_TSFC_KERNEL_CACHE_DIR PYOP2_CACHE_DIR Cheers, Tuomas On 07/14/2018 01:48 PM, Homolya, Miklós wrote: Hello, The relevant message seems to be this: A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. Firedrake compiles the high-level specification of your PDE into efficient binary code dynamically, just in time. It first generates C code, that calls the host C compiler to compile that. This step fails because your MPI does not support the fork() system call. Suggested workaround is to warm up your disk cache by first solving a small problem serially (or on a single node if that works), and then launch your parallel runs with warm cache, so Firedrake won't need to do further compilations. Regards, Miklos ________________________________ From: firedrake-bounces@imperial.ac.uk<mailto:firedrake-bounces@imperial.ac.uk> <firedrake-bounces@imperial.ac.uk><mailto:firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca><mailto:ambaierreinio@edu.uwaterloo.ca> Sent: 13 July 2018 19:15:51 To: firedrake Subject: [firedrake] Errors running Firedrake on multiple nodes Hello Firedrake team, I am trying to install Firedrake on a cluster and use mpi to run my Firedrake scripts on multiple nodes. I am running into some issues and was wondering if you would be able to help. The operating system on the cluster is CentOS 7.4. Both Python 3.5 and PETSc 3.9.0 are already loaded on the cluster via the environment modules tool. So, to install I am running the install script as follows: python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir --honour-pythonpath The install seems to go through successfully. The resultant installation seems to work when running on a single node, but I am getting errors when I run on multiple nodes. If I run my script as mpirun -np 8 python3 script.py and all 8 CPUs are located on the same node, then the script seems to run successfully. However, if the CPUs are distributed across multiple nodes, then I always get error messages. The general pattern to these error messages is that I will first get a PyOP2 caching error that looks like this: Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 20, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('WRITE'), <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 4, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('READ'), <class 'pyop2.sequential.Arg'>, (1,), dtype('float64'), Access('READ')) This error will then be followed by a bunch of "During handling of the above exception, another exception occurred" errors. I have attached the full output from one of these error messages. Do you have any idea what could be going wrong here? Please let me know if you need more information. Thanks, Aaron _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk<mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Hello again, It seems that warming the cache fixes the problem when the mesh resolution is small, but when the mesh resolution is increased the problem still occurs. As a MWE of this I have a file test.py that contains the following code: from firedrake import * from mpi4py import MPI set_log_level(logging.INFO) size = MPI.COMM_WORLD.Get_size() rank = MPI.COMM_WORLD.Get_rank() name = MPI.Get_processor_name() nn = 10 mesh = BoxMesh(nn, nn, nn, 1.0, 1.0, 1.0) DG = FunctionSpace(mesh, "DG", 1) expr = Expression("1.0", degree=1) f = project(expr, DG) print("Finished on process %d of %d on %s." % (rank, size, name)) To test this I run a Slurm job on 2 nodes with 1 CPU per node, so that "mpirun -np 1 python3 test.py" will run the script on one node only, whereas "-np 2" will run the script on two nodes. Now, with nn=10, if I run the following in sequence, mpirun -np 2 python3 test.py mpirun -np 1 python3 test.py mpirun -np 2 python3 test.py the first mpirun will fail as expected, but the last two both succeed. However, doing the same thing but with nn=40, both "-np 2" runs will fail. I have attached the output for both cases. Is what I am doing to warm the cache correct or is there a different way I should be doing it that will work for larger meshes? Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> Sent: July 14, 2018 2:17 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello Miklos and Tuomas, Thank you for the replies. I am not sure what is going on with the MPI fork warning. This warning message is always shown as soon as I import firedrake. It will show up no matter how many cores or nodes I am running on, and even if I run the script without using mpirun, it still shows up. I have just been ignoring it. Regarding your advice about warming the cache and/or changing the cache directories, I will try doing both of these things. I will let you know what happens. Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Tuomas Karna <tuomas.karna@gmail.com> Sent: July 14, 2018 1:04:39 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hi Miklos and Aaron, I agree that this looks like a compilation/cache issue. For what it's worth, however, on some linux clusters I do see the MPI fork warning but it does not affect firedrake; compilation works just fine in parallel. It might be worth trying to point the cache directories to a different location. I think they are set with these env variables: FIREDRAKE_TSFC_KERNEL_CACHE_DIR PYOP2_CACHE_DIR Cheers, Tuomas On 07/14/2018 01:48 PM, Homolya, Miklós wrote: Hello, The relevant message seems to be this: A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. Firedrake compiles the high-level specification of your PDE into efficient binary code dynamically, just in time. It first generates C code, that calls the host C compiler to compile that. This step fails because your MPI does not support the fork() system call. Suggested workaround is to warm up your disk cache by first solving a small problem serially (or on a single node if that works), and then launch your parallel runs with warm cache, so Firedrake won't need to do further compilations. Regards, Miklos ________________________________ From: firedrake-bounces@imperial.ac.uk<mailto:firedrake-bounces@imperial.ac.uk> <firedrake-bounces@imperial.ac.uk><mailto:firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca><mailto:ambaierreinio@edu.uwaterloo.ca> Sent: 13 July 2018 19:15:51 To: firedrake Subject: [firedrake] Errors running Firedrake on multiple nodes Hello Firedrake team, I am trying to install Firedrake on a cluster and use mpi to run my Firedrake scripts on multiple nodes. I am running into some issues and was wondering if you would be able to help. The operating system on the cluster is CentOS 7.4. Both Python 3.5 and PETSc 3.9.0 are already loaded on the cluster via the environment modules tool. So, to install I am running the install script as follows: python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir --honour-pythonpath The install seems to go through successfully. The resultant installation seems to work when running on a single node, but I am getting errors when I run on multiple nodes. If I run my script as mpirun -np 8 python3 script.py and all 8 CPUs are located on the same node, then the script seems to run successfully. However, if the CPUs are distributed across multiple nodes, then I always get error messages. The general pattern to these error messages is that I will first get a PyOP2 caching error that looks like this: Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 20, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('WRITE'), <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 4, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('READ'), <class 'pyop2.sequential.Arg'>, (1,), dtype('float64'), Access('READ')) This error will then be followed by a bunch of "During handling of the above exception, another exception occurred" errors. I have attached the full output from one of these error messages. Do you have any idea what could be going wrong here? Please let me know if you need more information. Thanks, Aaron _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk<mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Hello, Has anybody had a chance to look at this? Being able to use multi-node Firedrake would greatly accelerate my work, and I would be grateful for any suggestions. Aaron ________________________________ From: Aaron Matthew Baier-Reinio Sent: July 15, 2018 9:24:33 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello again, It seems that warming the cache fixes the problem when the mesh resolution is small, but when the mesh resolution is increased the problem still occurs. As a MWE of this I have a file test.py that contains the following code: from firedrake import * from mpi4py import MPI set_log_level(logging.INFO) size = MPI.COMM_WORLD.Get_size() rank = MPI.COMM_WORLD.Get_rank() name = MPI.Get_processor_name() nn = 10 mesh = BoxMesh(nn, nn, nn, 1.0, 1.0, 1.0) DG = FunctionSpace(mesh, "DG", 1) expr = Expression("1.0", degree=1) f = project(expr, DG) print("Finished on process %d of %d on %s." % (rank, size, name)) To test this I run a Slurm job on 2 nodes with 1 CPU per node, so that "mpirun -np 1 python3 test.py" will run the script on one node only, whereas "-np 2" will run the script on two nodes. Now, with nn=10, if I run the following in sequence, mpirun -np 2 python3 test.py mpirun -np 1 python3 test.py mpirun -np 2 python3 test.py the first mpirun will fail as expected, but the last two both succeed. However, doing the same thing but with nn=40, both "-np 2" runs will fail. I have attached the output for both cases. Is what I am doing to warm the cache correct or is there a different way I should be doing it that will work for larger meshes? Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> Sent: July 14, 2018 2:17 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello Miklos and Tuomas, Thank you for the replies. I am not sure what is going on with the MPI fork warning. This warning message is always shown as soon as I import firedrake. It will show up no matter how many cores or nodes I am running on, and even if I run the script without using mpirun, it still shows up. I have just been ignoring it. Regarding your advice about warming the cache and/or changing the cache directories, I will try doing both of these things. I will let you know what happens. Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Tuomas Karna <tuomas.karna@gmail.com> Sent: July 14, 2018 1:04:39 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hi Miklos and Aaron, I agree that this looks like a compilation/cache issue. For what it's worth, however, on some linux clusters I do see the MPI fork warning but it does not affect firedrake; compilation works just fine in parallel. It might be worth trying to point the cache directories to a different location. I think they are set with these env variables: FIREDRAKE_TSFC_KERNEL_CACHE_DIR PYOP2_CACHE_DIR Cheers, Tuomas On 07/14/2018 01:48 PM, Homolya, Miklós wrote: Hello, The relevant message seems to be this: A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. Firedrake compiles the high-level specification of your PDE into efficient binary code dynamically, just in time. It first generates C code, that calls the host C compiler to compile that. This step fails because your MPI does not support the fork() system call. Suggested workaround is to warm up your disk cache by first solving a small problem serially (or on a single node if that works), and then launch your parallel runs with warm cache, so Firedrake won't need to do further compilations. Regards, Miklos ________________________________ From: firedrake-bounces@imperial.ac.uk<mailto:firedrake-bounces@imperial.ac.uk> <firedrake-bounces@imperial.ac.uk><mailto:firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca><mailto:ambaierreinio@edu.uwaterloo.ca> Sent: 13 July 2018 19:15:51 To: firedrake Subject: [firedrake] Errors running Firedrake on multiple nodes Hello Firedrake team, I am trying to install Firedrake on a cluster and use mpi to run my Firedrake scripts on multiple nodes. I am running into some issues and was wondering if you would be able to help. The operating system on the cluster is CentOS 7.4. Both Python 3.5 and PETSc 3.9.0 are already loaded on the cluster via the environment modules tool. So, to install I am running the install script as follows: python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir --honour-pythonpath The install seems to go through successfully. The resultant installation seems to work when running on a single node, but I am getting errors when I run on multiple nodes. If I run my script as mpirun -np 8 python3 script.py and all 8 CPUs are located on the same node, then the script seems to run successfully. However, if the CPUs are distributed across multiple nodes, then I always get error messages. The general pattern to these error messages is that I will first get a PyOP2 caching error that looks like this: Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 20, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('WRITE'), <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 4, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('READ'), <class 'pyop2.sequential.Arg'>, (1,), dtype('float64'), Access('READ')) This error will then be followed by a bunch of "During handling of the above exception, another exception occurred" errors. I have attached the full output from one of these error messages. Do you have any idea what could be going wrong here? Please let me know if you need more information. Thanks, Aaron _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk<mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
In both cases what is happening is that the c compiler is failing to execute with what Python is seeing as a bad address exception. This is in a part of the code which is unaffected by mesh size, so I’m not convinced that mesh size is actually a cause. The two errors are different compiler calls (one is just checking the compiler version number while the other is the substantial compilation. In particular this means that in the n=10 case, the cache is definitely not warm. I guess this is the error from the first mpirun. In the case of the n=40 error, the exception is coming from a compiler flag check which happens before we even check the compiled code cache. This is appears to be a weak point in the warm cache workaround. I’m not sure what to do about that. David From: <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> Reply-To: firedrake <firedrake@imperial.ac.uk> Date: Monday, 16 July 2018 at 03:24 To: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello again, It seems that warming the cache fixes the problem when the mesh resolution is small, but when the mesh resolution is increased the problem still occurs. As a MWE of this I have a file test.py that contains the following code: from firedrake import * from mpi4py import MPI set_log_level(logging.INFO) size = MPI.COMM_WORLD.Get_size() rank = MPI.COMM_WORLD.Get_rank() name = MPI.Get_processor_name() nn = 10 mesh = BoxMesh(nn, nn, nn, 1.0, 1.0, 1.0) DG = FunctionSpace(mesh, "DG", 1) expr = Expression("1.0", degree=1) f = project(expr, DG) print("Finished on process %d of %d on %s." % (rank, size, name)) To test this I run a Slurm job on 2 nodes with 1 CPU per node, so that "mpirun -np 1 python3 test.py" will run the script on one node only, whereas "-np 2" will run the script on two nodes. Now, with nn=10, if I run the following in sequence, mpirun -np 2 python3 test.py mpirun -np 1 python3 test.py mpirun -np 2 python3 test.py the first mpirun will fail as expected, but the last two both succeed. However, doing the same thing but with nn=40, both "-np 2" runs will fail. I have attached the output for both cases. Is what I am doing to warm the cache correct or is there a different way I should be doing it that will work for larger meshes? Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> Sent: July 14, 2018 2:17 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello Miklos and Tuomas, Thank you for the replies. I am not sure what is going on with the MPI fork warning. This warning message is always shown as soon as I import firedrake. It will show up no matter how many cores or nodes I am running on, and even if I run the script without using mpirun, it still shows up. I have just been ignoring it. Regarding your advice about warming the cache and/or changing the cache directories, I will try doing both of these things. I will let you know what happens. Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Tuomas Karna <tuomas.karna@gmail.com> Sent: July 14, 2018 1:04:39 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hi Miklos and Aaron, I agree that this looks like a compilation/cache issue. For what it's worth, however, on some linux clusters I do see the MPI fork warning but it does not affect firedrake; compilation works just fine in parallel. It might be worth trying to point the cache directories to a different location. I think they are set with these env variables: FIREDRAKE_TSFC_KERNEL_CACHE_DIR PYOP2_CACHE_DIR Cheers, Tuomas On 07/14/2018 01:48 PM, Homolya, Miklós wrote: Hello, The relevant message seems to be this: A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. Firedrake compiles the high-level specification of your PDE into efficient binary code dynamically, just in time. It first generates C code, that calls the host C compiler to compile that. This step fails because your MPI does not support the fork() system call. Suggested workaround is to warm up your disk cache by first solving a small problem serially (or on a single node if that works), and then launch your parallel runs with warm cache, so Firedrake won't need to do further compilations. Regards, Miklos ________________________________ From: firedrake-bounces@imperial.ac.uk<mailto:firedrake-bounces@imperial.ac.uk> <firedrake-bounces@imperial.ac.uk><mailto:firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca><mailto:ambaierreinio@edu.uwaterloo.ca> Sent: 13 July 2018 19:15:51 To: firedrake Subject: [firedrake] Errors running Firedrake on multiple nodes Hello Firedrake team, I am trying to install Firedrake on a cluster and use mpi to run my Firedrake scripts on multiple nodes. I am running into some issues and was wondering if you would be able to help. The operating system on the cluster is CentOS 7.4. Both Python 3.5 and PETSc 3.9.0 are already loaded on the cluster via the environment modules tool. So, to install I am running the install script as follows: python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir --honour-pythonpath The install seems to go through successfully. The resultant installation seems to work when running on a single node, but I am getting errors when I run on multiple nodes. If I run my script as mpirun -np 8 python3 script.py and all 8 CPUs are located on the same node, then the script seems to run successfully. However, if the CPUs are distributed across multiple nodes, then I always get error messages. The general pattern to these error messages is that I will first get a PyOP2 caching error that looks like this: Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 20, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('WRITE'), <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 4, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('READ'), <class 'pyop2.sequential.Arg'>, (1,), dtype('float64'), Access('READ')) This error will then be followed by a bunch of "During handling of the above exception, another exception occurred" errors. I have attached the full output from one of these error messages. Do you have any idea what could be going wrong here? Please let me know if you need more information. Thanks, Aaron _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk<mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Thank you David for the breakdown and sorry to hear that this is seeing rather complicated. Even though you're not sure how best to fix this, might you suggest other tests that we could try to perhaps better understand the errors that are coming up? Francis ------------------ Francis Poulin Associate Dean, Undergraduate Studies Professor Department of Applied Mathematics University of Waterloo email: fpoulin@uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637 ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Ham, David A <david.ham@imperial.ac.uk> Sent: Thursday, July 19, 2018 4:56:04 AM To: firedrake Subject: Re: [firedrake] Errors running Firedrake on multiple nodes In both cases what is happening is that the c compiler is failing to execute with what Python is seeing as a bad address exception. This is in a part of the code which is unaffected by mesh size, so I’m not convinced that mesh size is actually a cause. The two errors are different compiler calls (one is just checking the compiler version number while the other is the substantial compilation. In particular this means that in the n=10 case, the cache is definitely not warm. I guess this is the error from the first mpirun. In the case of the n=40 error, the exception is coming from a compiler flag check which happens before we even check the compiled code cache. This is appears to be a weak point in the warm cache workaround. I’m not sure what to do about that. David From: <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> Reply-To: firedrake <firedrake@imperial.ac.uk> Date: Monday, 16 July 2018 at 03:24 To: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello again, It seems that warming the cache fixes the problem when the mesh resolution is small, but when the mesh resolution is increased the problem still occurs. As a MWE of this I have a file test.py that contains the following code: from firedrake import * from mpi4py import MPI set_log_level(logging.INFO) size = MPI.COMM_WORLD.Get_size() rank = MPI.COMM_WORLD.Get_rank() name = MPI.Get_processor_name() nn = 10 mesh = BoxMesh(nn, nn, nn, 1.0, 1.0, 1.0) DG = FunctionSpace(mesh, "DG", 1) expr = Expression("1.0", degree=1) f = project(expr, DG) print("Finished on process %d of %d on %s." % (rank, size, name)) To test this I run a Slurm job on 2 nodes with 1 CPU per node, so that "mpirun -np 1 python3 test.py" will run the script on one node only, whereas "-np 2" will run the script on two nodes. Now, with nn=10, if I run the following in sequence, mpirun -np 2 python3 test.py mpirun -np 1 python3 test.py mpirun -np 2 python3 test.py the first mpirun will fail as expected, but the last two both succeed. However, doing the same thing but with nn=40, both "-np 2" runs will fail. I have attached the output for both cases. Is what I am doing to warm the cache correct or is there a different way I should be doing it that will work for larger meshes? Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> Sent: July 14, 2018 2:17 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello Miklos and Tuomas, Thank you for the replies. I am not sure what is going on with the MPI fork warning. This warning message is always shown as soon as I import firedrake. It will show up no matter how many cores or nodes I am running on, and even if I run the script without using mpirun, it still shows up. I have just been ignoring it. Regarding your advice about warming the cache and/or changing the cache directories, I will try doing both of these things. I will let you know what happens. Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Tuomas Karna <tuomas.karna@gmail.com> Sent: July 14, 2018 1:04:39 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hi Miklos and Aaron, I agree that this looks like a compilation/cache issue. For what it's worth, however, on some linux clusters I do see the MPI fork warning but it does not affect firedrake; compilation works just fine in parallel. It might be worth trying to point the cache directories to a different location. I think they are set with these env variables: FIREDRAKE_TSFC_KERNEL_CACHE_DIR PYOP2_CACHE_DIR Cheers, Tuomas On 07/14/2018 01:48 PM, Homolya, Miklós wrote: Hello, The relevant message seems to be this: A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. Firedrake compiles the high-level specification of your PDE into efficient binary code dynamically, just in time. It first generates C code, that calls the host C compiler to compile that. This step fails because your MPI does not support the fork() system call. Suggested workaround is to warm up your disk cache by first solving a small problem serially (or on a single node if that works), and then launch your parallel runs with warm cache, so Firedrake won't need to do further compilations. Regards, Miklos ________________________________ From: firedrake-bounces@imperial.ac.uk<mailto:firedrake-bounces@imperial.ac.uk> <firedrake-bounces@imperial.ac.uk><mailto:firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca><mailto:ambaierreinio@edu.uwaterloo.ca> Sent: 13 July 2018 19:15:51 To: firedrake Subject: [firedrake] Errors running Firedrake on multiple nodes Hello Firedrake team, I am trying to install Firedrake on a cluster and use mpi to run my Firedrake scripts on multiple nodes. I am running into some issues and was wondering if you would be able to help. The operating system on the cluster is CentOS 7.4. Both Python 3.5 and PETSc 3.9.0 are already loaded on the cluster via the environment modules tool. So, to install I am running the install script as follows: python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir --honour-pythonpath The install seems to go through successfully. The resultant installation seems to work when running on a single node, but I am getting errors when I run on multiple nodes. If I run my script as mpirun -np 8 python3 script.py and all 8 CPUs are located on the same node, then the script seems to run successfully. However, if the CPUs are distributed across multiple nodes, then I always get error messages. The general pattern to these error messages is that I will first get a PyOP2 caching error that looks like this: Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 20, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('WRITE'), <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 4, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('READ'), <class 'pyop2.sequential.Arg'>, (1,), dtype('float64'), Access('READ')) This error will then be followed by a bunch of "During handling of the above exception, another exception occurred" errors. I have attached the full output from one of these error messages. Do you have any idea what could be going wrong here? Please let me know if you need more information. Thanks, Aaron _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk<mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Hi Francis, It seems to be the case that forking compilers during mpi runs on your cluster is unreliable. This can be an issue with some MPI setups. You can attempt to get to the bottom of it but that will be a deep dive into the MPI systems on your cluster which has little, if anything, to do with Firedrake. There are two potential fixes to your current problems. One is to modify PyOP2 to avoid executing compilers until after the cache miss occurs. That would probably make the warm cache workaround work for you. The other, probably more robust solution, would be to add to PyOP2 the possibility to call llvm as a library and therefore to avoid the fork call at all. This is a lot more work. I’m not sure if we get some of this for free when we move to the loopy backend. Regards, David From: <firedrake-bounces@imperial.ac.uk> on behalf of Francis Poulin <fpoulin@uwaterloo.ca> Reply-To: firedrake <firedrake@imperial.ac.uk> Date: Thursday, 19 July 2018 at 10:07 To: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Thank you David for the breakdown and sorry to hear that this is seeing rather complicated. Even though you're not sure how best to fix this, might you suggest other tests that we could try to perhaps better understand the errors that are coming up? Francis ------------------ Francis Poulin Associate Dean, Undergraduate Studies Professor Department of Applied Mathematics University of Waterloo email: fpoulin@uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637 ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Ham, David A <david.ham@imperial.ac.uk> Sent: Thursday, July 19, 2018 4:56:04 AM To: firedrake Subject: Re: [firedrake] Errors running Firedrake on multiple nodes In both cases what is happening is that the c compiler is failing to execute with what Python is seeing as a bad address exception. This is in a part of the code which is unaffected by mesh size, so I’m not convinced that mesh size is actually a cause. The two errors are different compiler calls (one is just checking the compiler version number while the other is the substantial compilation. In particular this means that in the n=10 case, the cache is definitely not warm. I guess this is the error from the first mpirun. In the case of the n=40 error, the exception is coming from a compiler flag check which happens before we even check the compiled code cache. This is appears to be a weak point in the warm cache workaround. I’m not sure what to do about that. David From: <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> Reply-To: firedrake <firedrake@imperial.ac.uk> Date: Monday, 16 July 2018 at 03:24 To: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello again, It seems that warming the cache fixes the problem when the mesh resolution is small, but when the mesh resolution is increased the problem still occurs. As a MWE of this I have a file test.py that contains the following code: from firedrake import * from mpi4py import MPI set_log_level(logging.INFO) size = MPI.COMM_WORLD.Get_size() rank = MPI.COMM_WORLD.Get_rank() name = MPI.Get_processor_name() nn = 10 mesh = BoxMesh(nn, nn, nn, 1.0, 1.0, 1.0) DG = FunctionSpace(mesh, "DG", 1) expr = Expression("1.0", degree=1) f = project(expr, DG) print("Finished on process %d of %d on %s." % (rank, size, name)) To test this I run a Slurm job on 2 nodes with 1 CPU per node, so that "mpirun -np 1 python3 test.py" will run the script on one node only, whereas "-np 2" will run the script on two nodes. Now, with nn=10, if I run the following in sequence, mpirun -np 2 python3 test.py mpirun -np 1 python3 test.py mpirun -np 2 python3 test.py the first mpirun will fail as expected, but the last two both succeed. However, doing the same thing but with nn=40, both "-np 2" runs will fail. I have attached the output for both cases. Is what I am doing to warm the cache correct or is there a different way I should be doing it that will work for larger meshes? Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca> Sent: July 14, 2018 2:17 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hello Miklos and Tuomas, Thank you for the replies. I am not sure what is going on with the MPI fork warning. This warning message is always shown as soon as I import firedrake. It will show up no matter how many cores or nodes I am running on, and even if I run the script without using mpirun, it still shows up. I have just been ignoring it. Regarding your advice about warming the cache and/or changing the cache directories, I will try doing both of these things. I will let you know what happens. Aaron ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Tuomas Karna <tuomas.karna@gmail.com> Sent: July 14, 2018 1:04:39 PM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes Hi Miklos and Aaron, I agree that this looks like a compilation/cache issue. For what it's worth, however, on some linux clusters I do see the MPI fork warning but it does not affect firedrake; compilation works just fine in parallel. It might be worth trying to point the cache directories to a different location. I think they are set with these env variables: FIREDRAKE_TSFC_KERNEL_CACHE_DIR PYOP2_CACHE_DIR Cheers, Tuomas On 07/14/2018 01:48 PM, Homolya, Miklós wrote: Hello, The relevant message seems to be this: A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. Firedrake compiles the high-level specification of your PDE into efficient binary code dynamically, just in time. It first generates C code, that calls the host C compiler to compile that. This step fails because your MPI does not support the fork() system call. Suggested workaround is to warm up your disk cache by first solving a small problem serially (or on a single node if that works), and then launch your parallel runs with warm cache, so Firedrake won't need to do further compilations. Regards, Miklos ________________________________ From: firedrake-bounces@imperial.ac.uk<mailto:firedrake-bounces@imperial.ac.uk> <firedrake-bounces@imperial.ac.uk><mailto:firedrake-bounces@imperial.ac.uk> on behalf of Aaron Matthew Baier-Reinio <ambaierreinio@edu.uwaterloo.ca><mailto:ambaierreinio@edu.uwaterloo.ca> Sent: 13 July 2018 19:15:51 To: firedrake Subject: [firedrake] Errors running Firedrake on multiple nodes Hello Firedrake team, I am trying to install Firedrake on a cluster and use mpi to run my Firedrake scripts on multiple nodes. I am running into some issues and was wondering if you would be able to help. The operating system on the cluster is CentOS 7.4. Both Python 3.5 and PETSc 3.9.0 are already loaded on the cluster via the environment modules tool. So, to install I am running the install script as follows: python3 firedrake-install --disable-ssh --no-package-manager --honour-petsc-dir --honour-pythonpath The install seems to go through successfully. The resultant installation seems to work when running on a single node, but I am getting errors when I run on multiple nodes. If I run my script as mpirun -np 8 python3 script.py and all 8 CPUs are located on the same node, then the script seems to run successfully. However, if the CPUs are distributed across multiple nodes, then I always get error messages. The general pattern to these error messages is that I will first get a PyOP2 caching error that looks like this: Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 20, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('WRITE'), <class 'pyop2.sequential.Arg'>, (3,), dtype('float64'), 4, (<class 'pyop2.base.IterationIndex'>, 0), None, Access('READ'), <class 'pyop2.sequential.Arg'>, (1,), dtype('float64'), Access('READ')) This error will then be followed by a bunch of "During handling of the above exception, another exception occurred" errors. I have attached the full output from one of these error messages. Do you have any idea what could be going wrong here? Please let me know if you need more information. Thanks, Aaron _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk<mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
On 19/07/18 11:09, Ham, David A wrote:
Hi Francis,
It seems to be the case that forking compilers during mpi runs on your cluster is unreliable. This can be an issue with some MPI setups. You can attempt to get to the bottom of it but that will be a deep dive into the MPI systems on your cluster which has little, if anything, to do with Firedrake.
There are two potential fixes to your current problems. One is to modify PyOP2 to avoid executing compilers until after the cache miss occurs. That would probably make the warm cache workaround work for you.
The other, probably more robust solution, would be to add to PyOP2 the possibility to call llvm as a library and therefore to avoid the fork call at all. This is a lot more work. I’m not sure if we get some of this for free when we move to the loopy backend.
A third option, which avoids the MPI issue, is to do the following: 1. Before calling MPI_Init, fork() a child process. This process will be able to call fork() later. 2. Now intercept all subprocess calls in MPI processes and forward them to this child process, that can actually do forking. Andreas wrote something to do that here: https://github.com/inducer/pytools/blob/master/pytools/prefork.py One has to be a little careful to carefully audit all the places where we might launch a subprocess. Cheers, Lawrence
Thank you Lawrence and David for your help with this. I will contact the people who maintain the cluster and ask about the MPI setup but don't imagine that will be too productive, unless there are more cases where MPI seems to breakdown. But I will try. As for the 3 possibilities that have been brought up, what would be the easiest/fastest? The code that Andreas wrote is there, which seems like it might be the easiest, but I can't say that I know what exactly to do with this. Which do you suggest we investigate? Cheers, Francis ------------------ Francis Poulin Associate Dean, Undergraduate Studies Professor Department of Applied Mathematics University of Waterloo email: fpoulin@uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637 ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk> Sent: Thursday, July 19, 2018 6:18:16 AM To: firedrake@imperial.ac.uk Subject: Re: [firedrake] Errors running Firedrake on multiple nodes On 19/07/18 11:09, Ham, David A wrote:
Hi Francis,
It seems to be the case that forking compilers during mpi runs on your cluster is unreliable. This can be an issue with some MPI setups. You can attempt to get to the bottom of it but that will be a deep dive into the MPI systems on your cluster which has little, if anything, to do with Firedrake.
There are two potential fixes to your current problems. One is to modify PyOP2 to avoid executing compilers until after the cache miss occurs. That would probably make the warm cache workaround work for you.
The other, probably more robust solution, would be to add to PyOP2 the possibility to call llvm as a library and therefore to avoid the fork call at all. This is a lot more work. I’m not sure if we get some of this for free when we move to the loopy backend.
A third option, which avoids the MPI issue, is to do the following: 1. Before calling MPI_Init, fork() a child process. This process will be able to call fork() later. 2. Now intercept all subprocess calls in MPI processes and forward them to this child process, that can actually do forking. Andreas wrote something to do that here: https://github.com/inducer/pytools/blob/master/pytools/prefork.py One has to be a little careful to carefully audit all the places where we might launch a subprocess. Cheers, Lawrence _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
participants (6)
-
Aaron Matthew Baier-Reinio
-
Francis Poulin
-
Ham, David A
-
Homolya, Miklós
-
Lawrence Mitchell
-
Tuomas Karna