Dear firedrakers, I now re-ran my code on up to 1536 cores on ARCHER, but I get a problem when I try to project an expression onto a DG0 function space on an extruded grid. The full (very large) log is here https://gist.github.com/eikehmueller/83a5fc139e1fedb5306c but as far as I can tell the following crashes: r_p.project(expression,solver_parameters={'ksp_type':'cg','pc_type':'jacobi’}) and here is the relevant part of the trace that I attempted to reconstruct: File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/function.py", line 157, in project return projection.project(b, self, *args, **kwargs) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/projection.py", line 94, in project […] solving_utils.check_snes_convergence(self.snes) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/variational_solver.py", line 163, in solve File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py", line 199, in wrapper %s""" % (snes.getIterationNumber(), msg)) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/solving_utils.py", line 62, in check_snes_convergence return f(*args, **kwargs) RuntimeError: Nonlinear solve failed to converge after 1 nonlinear iterations. It does work fine on smaller processor numbers. Maybe the PETSc integers overflow again, the number of cells is 5242880 x 64 = 335544320 ~ 2^{28}, which is not too far from 2^{32}, but I thought I check in case you’ve seen something similar before. I thought I had managed to run problems of this size in the past (i.e. earlier this year). Thanks, Eike