[0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done [0] pyop2:INFO Compiling wrapper... [0] pyop2:INFO Compiling wrapper...done =>> PBS: job killed: walltime 333 exceeded limit 300 aprun: Apid 12958237: Caught signal Terminated, sending to application [2]PETSC ERROR: [1]PETSC ERROR: [3]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: [1]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [2]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ Try option -start_in_debugger or -on_error_attach_debugger Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: [3]PETSC ERROR: [1]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: [5]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [4]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [5]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [7]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [6]PETSC ERROR: ------------------------------------------------------------------------ [8]PETSC ERROR: [7]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ [9]PETSC ERROR: [8]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end Try option -start_in_debugger or -on_error_attach_debugger [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [8]PETSC ERROR: [10]PETSC ERROR: [0]PETSC ERROR: [11]PETSC ERROR: ------------------------------------------------------------------------ [9]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [9]PETSC ERROR: ------------------------------------------------------------------------ [10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [10]PETSC ERROR: ------------------------------------------------------------------------ [11]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [11]PETSC ERROR: [13]PETSC ERROR: [12]PETSC ERROR: [14]PETSC ERROR: ------------------------------------------------------------------------ ------------------------------------------------------------------------ [15]PETSC ERROR: [13]PETSC ERROR: [12]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [13]PETSC ERROR: [12]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger Try option -start_in_debugger or -on_error_attach_debugger [13]PETSC ERROR: [12]PETSC ERROR: [16]PETSC ERROR: ------------------------------------------------------------------------ [17]PETSC ERROR: [14]PETSC ERROR: ------------------------------------------------------------------------ Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [15]PETSC ERROR: [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [14]PETSC ERROR: [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [15]PETSC ERROR: [18]PETSC ERROR: ------------------------------------------------------------------------ [16]PETSC ERROR: [19]PETSC ERROR: ------------------------------------------------------------------------ Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [17]PETSC ERROR: [16]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [16]PETSC ERROR: [17]PETSC ERROR: ------------------------------------------------------------------------ Try option -start_in_debugger or -on_error_attach_debugger [18]PETSC ERROR: [17]PETSC ERROR: [20]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end ------------------------------------------------------------------------ [18]PETSC ERROR: [19]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [18]PETSC ERROR: [19]PETSC ERROR: [21]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [19]PETSC ERROR: ------------------------------------------------------------------------ [20]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [20]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [20]PETSC ERROR: [22]PETSC ERROR: ------------------------------------------------------------------------ [21]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [21]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [21]PETSC ERROR: [23]PETSC ERROR: ------------------------------------------------------------------------ [22]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [22]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [22]PETSC ERROR: ------------------------------------------------------------------------ [23]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [23]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [23]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: [9]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [9]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [13]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [13]PETSC ERROR: to get more information on the crash. or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [21]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [21]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [21]PETSC ERROR: to get more information on the crash. or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 9 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 9 or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [14]PETSC ERROR: [7]PETSC ERROR: [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [17]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind configure using --with-debugging=yes, recompile, link, and run or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [12]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [11]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [10]PETSC ERROR: [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [6]PETSC ERROR: [3]PETSC ERROR: [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 21 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 21 configure using --with-debugging=yes, recompile, link, and run or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run configure using --with-debugging=yes, recompile, link, and run configure using --with-debugging=yes, recompile, link, and run or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: [10]PETSC ERROR: to get more information on the crash. to get more information on the crash. [16]PETSC ERROR: [17]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors configure using --with-debugging=yes, recompile, link, and run to get more information on the crash. [15]PETSC ERROR: [16]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run to get more information on the crash. or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [15]PETSC ERROR: [7]PETSC ERROR: to get more information on the crash. configure using --with-debugging=yes, recompile, link, and run [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind to get more information on the crash. [14]PETSC ERROR: [18]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors configure using --with-debugging=yes, recompile, link, and run [18]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [18]PETSC ERROR: [14]PETSC ERROR: to get more information on the crash. to get more information on the crash. or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [5]PETSC ERROR: Rank 1 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run Rank 10 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10 [5]PETSC ERROR: to get more information on the crash. Rank 17 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 17 Rank 16 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 16 Rank 0 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [2]PETSC ERROR: Rank 15 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15 to get more information on the crash. [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 7 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 7 [11]PETSC ERROR: [3]PETSC ERROR: [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run configure using --with-debugging=yes, recompile, link, and run configure using --with-debugging=yes, recompile, link, and run [4]PETSC ERROR: [3]PETSC ERROR: to get more information on the crash. to get more information on the crash. [11]PETSC ERROR: Rank 18 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 18 Rank 14 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14 to get more information on the crash. Rank 13 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 13 or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [8]PETSC ERROR: [6]PETSC ERROR: to get more information on the crash. configure using --with-debugging=yes, recompile, link, and run Rank 5 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5 [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind to get more information on the crash. Rank 2 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 [20]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [20]PETSC ERROR: [23]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [23]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run configure using --with-debugging=yes, recompile, link, and run [20]PETSC ERROR: to get more information on the crash. [23]PETSC ERROR: [19]PETSC ERROR: to get more information on the crash. Rank 4 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4 or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 3 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 Rank 11 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 11 [19]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [19]PETSC ERROR: to get more information on the crash. [22]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 8 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 8 [22]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run Rank 6 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6 [22]PETSC ERROR: to get more information on the crash. or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [12]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [12]PETSC ERROR: to get more information on the crash. Rank 20 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 20 Rank 23 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 23 Rank 19 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 19 Rank 22 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 22 Rank 12 [Sun Feb 15 22:16:51 2015] [c1-3c2s13n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 12 _pmiu_daemon(SIGCHLD): [NID 04981] [c1-3c2s13n1] [Sun Feb 15 22:16:51 2015] PE RANK 9 exit signal Aborted