Projection on DG0 space fails on large core counts

30 May 2015

      Dear firedrakers,

I now re-ran my code on up to 1536 cores on ARCHER, but I get a problem when I try to project an expression onto a DG0 function space on an extruded grid.

The full (very large) log is here https://gist.github.com/eikehmueller/83a5fc139e1fedb5306c but as far as I can tell

the following crashes:

r_p.project(expression,solver_parameters={'ksp_type':'cg','pc_type':'jacobi’})

and here is the relevant part of the trace that I attempted to reconstruct:

  File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/function.py", line 157, in project
return projection.project(b, self, *args, **kwargs)
  File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/projection.py", line 94, in project
[…]
    solving_utils.check_snes_convergence(self.snes)
  File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/variational_solver.py", line 163, in solve
  File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py", line 199, in wrapper
    %s""" % (snes.getIterationNumber(), msg))
  File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/solving_utils.py", line 62, in check_snes_convergence
    return f(*args, **kwargs)
RuntimeError: Nonlinear solve failed to converge after 1 nonlinear iterations.

It does work fine on smaller processor numbers. Maybe the PETSc integers overflow again, the number of cells is 5242880 x 64 = 335544320 ~ 2^{28}, which is not too far from 2^{32}, but I thought I check in case you’ve seen something similar before. I thought I had managed to run problems of this size in the past (i.e. earlier this year).

Thanks,

Eike

Eike Mueller

Lawrence Mitchell

tags

participants (2)