Hello Everyone, Thanks Colin and David for the helpful advice. David: I'm glad to know that things should scale much better than this. On my laptop that only has 4 cores, I did a some tests to see how long it took to output one day (100 time steps). cpu = 4: time: initially 24, then 25 and 26. Approximately 25. memory: initially 6%, then 8% cpu = 2: time: about 36 between outputs cpu = 1: time: about 64 between outputs Going from 1 to 2 cores is pretty efficient, about 90%, but then going to 4 we get 67%. Maybe it's because of the architecture. I'm not sure. Colin: Following what you set up initially we are choosing the following function spaces. Vdg = FunctionSpace(mesh,"DG",1) # DG elements for Potential Vorticity (PV) Vcg = FunctionSpace(mesh,"CG",1) # CG elements for Streamfunction Vu = VectorFunctionSpace(mesh,"DG",1, dim=2) # DG elements for velocity So velocity is already a DG space and the same degree at the PV space. Is there something here that I'm not getting? As for the projection vs interpolate, we have the following two lines that project. v = Function(Vu, name='velocity').project(gradperp_h(psi0)) v.project(gradperp_h(psi0)) It seems easy enough to change project to interpolate and I will give that a try. I can appreciate that maybe ilu is using a lot of memory but should it be using more memory as time passes? Significantly more memory. I would think that it's not allocating anymore arrays and it's solving the same problem at every step which does not change much in the first 40 days. Is there a reason to think why at day 32 suddenly uses so much more memory? I am doing the cpu = 4 test case right now and I do see a steady increase in the percentage of memory usage: 6, 8 and now 8.7. I have 16 GB of memory and the increase seems very dramatic to me but I don't have that much experience using PETSc so maybe this is to be expected. If there is a possibility to use AMG to speed things up and improve the efficiency I am very keen to try it. I presume this requires using a different branch of the gitrepo? Best regards, Francis ------------------ Francis Poulin Associate Professor Department of Applied Mathematics University of Waterloo email: fpoulin@uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637 ________________________________ From: firedrake-bounces@imperial.ac.uk [firedrake-bounces@imperial.ac.uk] on behalf of Colin Cotter [colin.cotter@imperial.ac.uk] Sent: Sunday, May 22, 2016 2:11 PM To: firedrake Subject: Re: [firedrake] aborted run ... in which case we should get you set up with AMG for the PV inversion as well. all the best --cjc On 22 May 2016 at 11:34, Ham, David A <david.ham@imperial.ac.uk<mailto:david.ham@imperial.ac.uk>> wrote: Additionally, that's something like half a million cells, so you should be able to go faster by using more cores. Depending on your discretisation, 10-20 cores probably makes sense for that problem. Regards, David On Sun, 22 May 2016 at 08:24 Cotter, Colin J <colin.cotter@imperial.ac.uk<mailto:colin.cotter@imperial.ac.uk>> wrote: Hi Francis, This is because the solver inside the projection is using ilu and it is running out of memory. More scalable solvers are available but in this case I would suggest making the velocity space into a DG vector function space large enough to contain your grad perp (same degree as the pv space works) and then using interpolate instead of project, which produces the same result mathematically but can then be computed locally. This wasn't previously possible which is why we put project. All the best Cjc On Sunday, 22 May 2016, Francis Poulin <fpoulin@uwaterloo.ca<mailto:fpoulin@uwaterloo.ca>> wrote: Hello, First, sorry to trouble you on the weekend, if you are reading this on the weekend that is. Second, Jemma and I have had some success with our 3D QG model. You can seem the result of the instability of a barotropic jet a very course 50x50x50 run at the following link. The movie needs a lot of fixing up but energy and enstrophy are nearly conserved so somethings are going pretty well. https://www.youtube.com/watch?v=CbqdA3a9DZg I decide to try a finer resolution case, 100x100x50, in hopes of having a better data set to work with. I am using a time step of 0.01 and it does 1 time unit on 2 cores on my laptop in over an hour. Not sure if that is very fast or not but it was working away up until it died. Below you see the last few outputs of the energies before things went all pear shaped. I have a fixed time step and so can believe that maybe the CFL condition was violated but for a 2D case I did a 200x200 run with the same time step so I'm not convinced that the CFL condition is violated. Given the output, does anyone have any ideas on what might have gone wrong? It would be great to restart from the last output and see whether this happens every time. Unfortunately we haven't gotten to that stage yet but we really should do that next. Cheers, Francis ... 33.98 16.504104156 39.1507283386 55.6548324946 33.3509321074 33.99 16.504103109 39.1507164876 55.6548195966 33.3509288412 34.0 16.5041020578 39.1507045844 55.6548066422 33.3509255645 Traceback (most recent call last): File "qg3d_jet.py", line 151, in <module> v.project(gradperp_h(psi0)) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/function.py", line 277, in project return projection.project(b, self, *args, **kwargs) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/projection.py", line 101, in project form_compiler_parameters=form_compiler_parameters) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/solving.py", line 118, in solve _solve_varproblem(*args, **kwargs) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/solving.py", line 145, in _solve_varproblem solver.solve() File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/variational_solver.py", line 185, in solve self.snes.solve(None, v) File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:168132) Traceback (most recent call last): File "qg3d_jet.py", line 151, in <module> v.project(gradperp_h(psi0)) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/function.py", line 277, in project return projection.project(b, self, *args, **kwargs) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/projection.py", line 101, in project form_compiler_parameters=form_compiler_parameters) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/solving.py", line 118, in solve _solve_varproblem(*args, **kwargs) File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/solving.py", line 145, in _solve_varproblem solver.solve() File "/home/fpoulin/software/firedrake/local/lib/python2.7/site-packages/firedrake/variational_solver.py", line 185, in solve self.snes.solve(None, v) File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:168132) petsc4py.PETSc.Errorpetsc4py.PETSc.Error: error code 1 [0] SNESSolve() line 4008 in /tmp/pip-EZZoo0-build/src/snes/interface/snes.c [0] SNESSolve_KSPONLY() line 40 in /tmp/pip-EZZoo0-build/src/snes/impls/ksponly/ksponly.c [0] KSPSolve() line 600 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [0] KSPSetUpOnBlocks() line 220 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [0] PCSetUpOnBlocks() line 1001 in /tmp/pip-EZZoo0-build/src/ksp/pc/interface/precon.c [0] PCSetUpOnBlocks_BJacobi_Singleblock() line 650 in /tmp/pip-EZZoo0-build/src/ksp/pc/impls/bjacobi/bjacobi.c [0] KSPSetUp() line 390 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [0] PCSetUp() line 968 in /tmp/pip-EZZoo0-build/src/ksp/pc/interface/precon.c [0] PCSetUp_ILU() line 212 in /tmp/pip-EZZoo0-build/src/ksp/pc/impls/factor/ilu/ilu.c [0] MatILUFactorSymbolic() line 6594 in /tmp/pip-EZZoo0-build/src/mat/interface/matrix.c [0] MatILUFactorSymbolic_SeqBAIJ() line 370 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [0] MatILUFactorSymbolic_SeqBAIJ_ilu0() line 276 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [0] MatILUFactorSymbolic_SeqBAIJ_ilu0() line 276 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [0] Out of memory. Allocated: 0, Used by process: 2962321408 [0] Memory requested 584000046 : error code 1 [1] SNESSolve() line 4008 in /tmp/pip-EZZoo0-build/src/snes/interface/snes.c [1] SNESSolve_KSPONLY() line 40 in /tmp/pip-EZZoo0-build/src/snes/impls/ksponly/ksponly.c [1] KSPSolve() line 600 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [1] KSPSetUpOnBlocks() line 220 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [1] PCSetUpOnBlocks() line 1001 in /tmp/pip-EZZoo0-build/src/ksp/pc/interface/precon.c [1] PCSetUpOnBlocks_BJacobi_Singleblock() line 650 in /tmp/pip-EZZoo0-build/src/ksp/pc/impls/bjacobi/bjacobi.c [1] KSPSetUp() line 390 in /tmp/pip-EZZoo0-build/src/ksp/ksp/interface/itfunc.c [1] PCSetUp() line 968 in /tmp/pip-EZZoo0-build/src/ksp/pc/interface/precon.c [1] PCSetUp_ILU() line 212 in /tmp/pip-EZZoo0-build/src/ksp/pc/impls/factor/ilu/ilu.c [1] MatILUFactorSymbolic() line 6594 in /tmp/pip-EZZoo0-build/src/mat/interface/matrix.c [1] MatILUFactorSymbolic_SeqBAIJ() line 370 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [1] MatILUFactorSymbolic_SeqBAIJ_ilu0() line 276 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [1] MatILUFactorSymbolic_SeqBAIJ_ilu0() line 276 in /tmp/pip-EZZoo0-build/src/mat/impls/baij/seq/baijfact2.c [1] Out of memory. Allocated: 0, Used by process: 2943778816 [1] Memory requested 584000046 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 1. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- ------------------ Francis Poulin Associate Professor Department of Applied Mathematics University of Waterloo email: fpoulin@uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637<tel:%2B1%20519%20888%204567%20x32637> -- http://www.imperial.ac.uk/people/colin.cotter www.cambridge.org/9781107663916<http://www.cambridge.org/9781107663916> [http://assets.cambridge.org/97811076/63916/cover/9781107663916.jpg] -- http://www.imperial.ac.uk/people/colin.cotter www.cambridge.org/9781107663916<http://www.cambridge.org/9781107663916> [http://assets.cambridge.org/97811076/63916/cover/9781107663916.jpg]