Hello, First, sorry to trouble you on the weekend, if you are reading this on the weekend that is. Second, Jemma and I have had some success with our 3D QG model. You can seem the result of the instability of a barotropic jet a very course 50x50x50 run at the following link. The movie needs a lot of fixing up but energy and enstrophy are nearly conserved so somethings are going pretty well. https://www.youtube.com/watch?v=CbqdA3a9DZg I decide to try a finer resolution case, 100x100x50, in hopes of having a better data set to work with. I am using a time step of 0.01 and it does 1 time unit on 2 cores on my laptop in over an hour. Not sure if that is very fast or not but it was working away up until it died. Below you see the last few outputs of the energies before things went all pear shaped. I have a fixed time step and so can believe that maybe the CFL condition was violated but for a 2D case I did a 200x200 run with the same time step so I'm not convinced that the CFL condition is violated. Given the output, does anyone have any ideas on what might have gone wrong? It would be great to restart from the last output and see whether this happens every time. Unfortunately we haven't gotten to that stage yet but we really should do that next. Cheers, Francis ... 33.98 16.504104156 39.1507283386 55.6548324946 33.3509321074 33.99 16.504103109 39.1507164876 55.6548195966 33.3509288412 34.0 16.5041020578 39.1507045844 55.6548066422 33.3509255645 Traceback (most recent call last): File "qg3d_jet.py", line 151, in <module> v.project(gradperp_h(psi0)) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/function.py", line 277, in project return projection.project(b, self, *args, **kwargs) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/projection.py", line 101, in project form_compiler_parameters=form_compiler_parameters) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/solving.py", line 118, in solve _solve_varproblem(*args, **kwargs) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/solving.py", line 145, in _solve_varproblem solver.solve() File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/variational_solver.py", line 185, in solve self.snes.solve(None, v) File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:168132) Traceback (most recent call last): File "qg3d_jet.py", line 151, in <module> v.project(gradperp_h(psi0)) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/function.py", line 277, in project return projection.project(b, self, *args, **kwargs) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/projection.py", line 101, in project form_compiler_parameters=form_compiler_parameters) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/solving.py", line 118, in solve _solve_varproblem(*args, **kwargs) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/solving.py", line 145, in _solve_varproblem solver.solve() File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/variational_solver.py", line 185, in solve self.snes.solve(None, v) File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:168132) petsc4py.PETSc.Errorpetsc4py.PETSc.Error: error code 1 [0] SNESSolve() line 4008 in /tmp/pip-EZZoo0-build/src/snes/interface/snes.c [0] SNESSolve_KSPONLY() line 40 in /tmp/pip-EZZoo0-build/src/snes/impls/ksponly/ksponly.c [0] KSPSolve() line 600 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [0] KSPSetUpOnBlocks() line 220 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [0] PCSetUpOnBlocks() line 1001 in /tmp/pip-EZZoo0-build/src/ksp/pc/interface/precon.c [0] PCSetUpOnBlocks_BJacobi_Singleblock() line 650 in /tmp/pip-EZZoo0-build/src/ksp/pc/impls/bjacobi/bjacobi.c [0] KSPSetUp() line 390 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [0] PCSetUp() line 968 in /tmp/pip-EZZoo0-build/src/ksp/pc/interface/precon.c [0] PCSetUp_ILU() line 212 in /tmp/pip-EZZoo0-build/src/ksp/pc/impls/factor/ilu/ilu.c [0] MatILUFactorSymbolic() line 6594 in /tmp/pip-EZZoo0-build/src/mat/interface/matrix.c [0] MatILUFactorSymbolic_SeqBAIJ() line 370 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [0] MatILUFactorSymbolic_SeqBAIJ_ilu0() line 276 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [0] MatILUFactorSymbolic_SeqBAIJ_ilu0() line 276 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [0] Out of memory. Allocated: 0, Used by process: 2962321408 [0] Memory requested 584000046 : error code 1 [1] SNESSolve() line 4008 in /tmp/pip-EZZoo0-build/src/snes/interface/snes.c [1] SNESSolve_KSPONLY() line 40 in /tmp/pip-EZZoo0-build/src/snes/impls/ksponly/ksponly.c [1] KSPSolve() line 600 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [1] KSPSetUpOnBlocks() line 220 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [1] PCSetUpOnBlocks() line 1001 in /tmp/pip-EZZoo0-build/src/ksp/pc/interface/precon.c [1] PCSetUpOnBlocks_BJacobi_Singleblock() line 650 in /tmp/pip-EZZoo0-build/src/ksp/pc/impls/bjacobi/bjacobi.c [1] KSPSetUp() line 390 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [1] PCSetUp() line 968 in /tmp/pip-EZZoo0-build/src/ksp/pc/interface/precon.c [1] PCSetUp_ILU() line 212 in /tmp/pip-EZZoo0-build/src/ksp/pc/impls/factor/ilu/ilu.c [1] MatILUFactorSymbolic() line 6594 in /tmp/pip-EZZoo0-build/src/mat/interface/matrix.c [1] MatILUFactorSymbolic_SeqBAIJ() line 370 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [1] MatILUFactorSymbolic_SeqBAIJ_ilu0() line 276 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [1] MatILUFactorSymbolic_SeqBAIJ_ilu0() line 276 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [1] Out of memory. Allocated: 0, Used by process: 2943778816 [1] Memory requested 584000046 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 1. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- ------------------ Francis Poulin Associate Professor Department of Applied Mathematics University of Waterloo email: fpoulin@uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637