Dear all,

I'm wondering whether this isn't an error in a PETSc call after all, since it complains about a segfault. I provide callbacks to PETSc KSP solvers in my code, and for this I have to set the size of vectors that the KSP operates on. In the mixed preconditioner, I want it to operate on a vector which contains both the pressure and the velocity dofs. To count the total number of dofs I do this:

helmholtz.py: 60
        self.ndof_phi = self.V_pressure.dof_dset.size
        self.ndof_u = self.V_velocity.dof_dset.size
        self.ndof = self.ndof_phi+self.ndof_u

and then I set up the operator for the KSP like this:

helmholtz.py: 69
        op = PETSc.Mat().create()
        op.setSizes(((self.ndof, None), (self.ndof, None)))

In the preconditioner class I copy the dofs from the vectors encapsulated in the firedrake pressure- and velocity functions in and out and then call my matrix-free solver routine:

helmholtz.py: 365
        with self.phi_tmp.dat.vec as v:
            v.array[:] = x.array[:self.ndof_phi]
        with self.u_tmp.dat.vec as v:
            v.array[:] = x.array[self.ndof_phi:]
        self.solve(self.phi_tmp,self.u_tmp,
                   self.P_phi_tmp,self.P_u_tmp)
        with self.P_phi_tmp.dat.vec_ro as v:
            y.array[:self.ndof_phi] = v.array[:]
        with self.P_u_tmp.dat.vec_ro as v:
            y.array[self.ndof_phi:] = v.array[:]

Is this the correct way of doing it, in particular the use of self.V_pressure.dof_dset.size?
It's conceivable that it doesn't have problems for smaller grid sizes, and then for larger grid sizes it crashes on one processor where there is an out-of-bounds access, but it reports the compilation error on the master processor?

Thanks,

Eike

On 04/10/14 14:10, Florian Rathgeber wrote:

On 04/10/14 13:15, Eike Mueller wrote:

Dear firedrakers,

I tried to run the Helmholtz multigrid solver on higher core counts on
ARCHER. In certain configurations it works, but in others it crashes.

The number of fine grid cells is 20*4^(4+refcount_coarse).

These mesh sizes should all be manageable afaict, I have run on larger
meshes (up to ~60M cells afair).

In particular, I was not able to run on 48 cores (i.e. 2 full nodes)

1 node, 16 cores/node = 16 cores, refcount_coarse=5:  Runs
2 nores, 16 cores/node = 48 cores, refcount_coarse=4: hangs without
printing error message

I presume you mean 24 cores/node? How long did this run for? Was it
eventually killed by the scheduler?

2 nores, 16 cores/node = 48 cores, refcount_coarse=5: crashes with PETSc
error message

This isn't a PETSc error, it's only PETSc reporting the crash. The
actual issue is a compile error. Can you check whether
/work/n02/n02/eike//pyop2-cache/9b5f79130a429e4044a23dacb78191df.so or
.so.tmp exists?

2 nores, 16 cores/node = 48 cores, refcount_coarse=5: crashes with
different error message

This is actually the same error. Does
/work/n02/n02/eike//pyop2-cache/c8aac6eec48490d587593472c04a5557.so or
.so.tmp exist?

Output logs for all cases are attached, as well as the pbs scripts for
running on 48 cores.

The job script appears to be for a 32 core run. It looks legit, though
you wouldn't need to load the firedrake module given that you also load
the dependency modules and use your own PyOP2/Firedrake

I also tried running the poisson example in firedrake-benchmark in
different configurations:

1 node, 16 cores/node = 16 cores => works
2 nodes, 8 cores/node = 16 cores => works
2 nodes, 24 cores/node = 48 cores => crashes with PETSc error

This one is baffling: it appears that only ranks 24-47 are aborted. I
have absolutely no clue what is going on. Is this reproducible?

I did use the generic ARCHER firedrake installation in this case,
though. In the Helmholtz solver runs I used my own version of firedrake
and PyOP2 since I need the multigrid and local_par-loop branches.

This makes me suspicious that it is not a problem specific to my code,
but then I thought that you did run firedrake on relatively large core
counts before, didn't you?

I have run on up to 1536 cores for Poisson and Cahn-Hilliard.

The output of the Poisson benchmarks is also attached. 

Any ideas what might be going wrong and what is the best way for
debugging this?

I'm not entirely sure what to suggest about the run that just hangs
other than trying to pinpoint where exactly that happens.

For the compilation failures, I recall I had those sporadically and
could never figure out what exactly had gone wrong. Check what is in the
cache and also what the compiled code represents: and assembly or an
expression?

Florian

Thanks,

Eike

_______________________________________________
firedrake mailing list
firedrake@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/firedrake

-- 
Dr Eike Hermann Mueller
Research Associate (PostDoc)

Department of Mathematical Sciences
University of Bath
Bath BA2 7AY, United Kingdom

+44 1225 38 5803
e.mueller@bath.ac.uk
http://people.bath.ac.uk/em459/