(firedrake) [ambaierr@gra798 ManufacturedSolutionTests]$ mpirun -np 8 python3 Test3D_1.py -------------------------------------------------------------------------- A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. The process that invoked fork was: Local host: [[1073,1],3] (PID 2035) If you are *absolutely sure* that your application will successfully and correctly survive a call to fork(), you may disable this warning by setting the mpi_warn_on_fork MCA parameter to 0. -------------------------------------------------------------------------- [gra798:02017] 7 more processes have sent help message help-opal-runtime.txt / opal_init:warn-fork [gra798:02017] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 197, in __new__ return cls._cache_lookup(key) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 205, in _cache_lookup return cls._cache[key] KeyError: ('9d856de341689e4f60b3a9709f8cabd1', False, False, False, , (3,), dtype('float64'), 20, (, 0), None, Access('WRITE'), , (3,), dtype('float64'), 4, (, 0), None, Access('READ'), , (1,), dtype('float64'), Access('READ')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ambaierr/firedrake/src/PyOP2/pyop2/compilation.py", line 179, in compiler_version return Compiler.compiler_versions[self._cc] KeyError: 'mpicc' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "PETSc/petscsnes.pxi", line 265, in petsc4py.PETSc.SNES_Function File "/home/ambaierr/firedrake/src/firedrake/firedrake/solving_utils.py", line 436, in form_function with ctx._F.dat.vec_ro as v: File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.5.4/lib/python3.5/contextlib.py", line 59, in __enter__ return next(self.gen) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/petsc_base.py", line 331, in vec_context write=access is not base.READ) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/base.py", line 1460, in _force_evaluation _trace.evaluate(reads, writes) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/base.py", line 193, in evaluate comp._run() File "/home/ambaierr/firedrake/src/PyOP2/pyop2/base.py", line 3971, in _run return self.compute() File "/home/ambaierr/firedrake/src/PyOP2/pyop2/base.py", line 4012, in compute fun = self._jitmodule File "/home/ambaierr/firedrake/src/PyOP2/pyop2/utils.py", line 62, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/sequential.py", line 927, in _jitmodule pass_layer_arg=self._pass_layer_arg) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 199, in __new__ obj = make_obj() File "/home/ambaierr/firedrake/src/PyOP2/pyop2/caching.py", line 189, in make_obj obj.__init__(*args, **kwargs) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/sequential.py", line 756, in __init__ self.compile() File "/home/ambaierr/firedrake/src/PyOP2/pyop2/sequential.py", line 842, in compile comm=self.comm) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/compilation.py", line 434, in load compiler = LinuxCompiler(cppargs, ldargs, cpp=cpp, comm=comm) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/compilation.py", line 381, in __init__ cpp=cpp, comm=comm) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/compilation.py", line 170, in __init__ self._cppargs = cppargs + configuration['cflags'].split() + self.workaround_cflags File "/home/ambaierr/firedrake/src/PyOP2/pyop2/compilation.py", line 187, in workaround_cflags compiler, ver = self.compiler_version File "/home/ambaierr/firedrake/src/PyOP2/pyop2/compilation.py", line 181, in compiler_version ver = sniff_compiler_version(self._cc) File "/home/ambaierr/firedrake/src/PyOP2/pyop2/compilation.py", line 67, in sniff_compiler_version ver = subprocess.check_output([cc, "--version"]).decode("utf-8") File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.5.4/lib/python3.5/subprocess.py", line 316, in check_output **kwargs).stdout File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.5.4/lib/python3.5/subprocess.py", line 383, in run with Popen(*popenargs, **kwargs) as process: File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.5.4/lib/python3.5/subprocess.py", line 676, in __init__ restore_signals, start_new_session) File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.5.4/lib/python3.5/subprocess.py", line 1289, in _execute_child raise child_exception_type(errno_num, err_msg) OSError: [Errno 14] Bad address During handling of the above exception, another exception occurred: SystemError: returned a result with an error set -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 1. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [1]PETSC ERROR: to get more information on the crash. [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [2]PETSC ERROR: to get more information on the crash. [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [3]PETSC ERROR: to get more information on the crash. [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [4]PETSC ERROR: to get more information on the crash. [5]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [5]PETSC ERROR: to get more information on the crash. [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [6]PETSC ERROR: to get more information on the crash. [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [7]PETSC ERROR: to get more information on the crash. [gra798:02017] 7 more processes have sent help message help-mpi-api.txt / mpi-abort