Hi Lawrence, as an additional test I just tried to re-run PETSc’s ex6 on 24 cores with Euclid. Here it does seem to work, the command I use is: mpirun -n 24 ./ex6 -f -f helmholtz-sphere.dat -ksp_type cg -ksp_convergence_test skip -ksp_max_it 2 -ksp_monitor -pc_hypre_type boomeramg -pc_type hypre -pc_hypre_boomeramg_smooth_type Euclid -log_summary -ksp_view As a sanity check, is there a way of writing out the pressure matrix for the system that I currently solve with the fieldsplit preconditioner and run it through ex6 exactly like above? Thanks, Eike -- Dr Eike Hermann Mueller Lecturer in Scientific Computing Department of Mathematical Sciences University of Bath Bath BA2 7AY, United Kingdom +44 1225 38 6241 e.mueller@bath.ac.uk<mailto:e.mueller@bath.ac.uk> http://people.bath.ac.uk/em459/ On 13 May 2015, at 12:34, Eike Mueller <E.Mueller@bath.ac.uk<mailto:E.Mueller@bath.ac.uk>> wrote: Hi Lawrence, that's what I feared, an issue somewhere deep inside hypre/PETSc... I just reran sequentially on ARCHER with exactly the same setup as on my laptop, and then it does work. So maybe the problem is too large, I will try to reproduce the issue on my laptop. Thanks a lot anyway, Eike ________________________________________ From: firedrake-bounces@imperial.ac.uk<mailto:firedrake-bounces@imperial.ac.uk> <firedrake-bounces@imperial.ac.uk<mailto:firedrake-bounces@imperial.ac.uk>> on behalf of Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk<mailto:lawrence.mitchell@imperial.ac.uk>> Sent: 13 May 2015 11:21 To: firedrake@imperial.ac.uk<mailto:firedrake@imperial.ac.uk> Subject: Re: [firedrake] EUCLID BoomerAMG smoother on ARCHER -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 13/05/15 11:00, Eike Mueller wrote: Hi, I just tried to run the PETSc fieldsplit solver with the EUCLID smoother for the BoomerAMG preconditioner on ARCHER, and I get this error message: ============= error stack trace ==================== [18] ERROR: global index 112 not found in map Numbering_dhGlobalToLocal file= Numbering_dh.c line= 175 [18] called from: Factor_dhSolveSetup file= Factor_dh.c line= 555 [18] called from: Euclid_dhSetup file= Euclid_dh.c line= 282 [18] called from: HYPRE_EuclidSetup file= HYPRE_parcsr_Euclid.c line= 278 any idea what might cause this? I can run this without problems on my laptop. This looks like it's happened deep inside hypre itself. So I'm sort of out of ideas. Does this run, at the same scale, on your laptop successfully? Lawrence -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJVUyWlAAoJECOc1kQ8PEYvdz0H/11HfQy0p7LO1DpGyoXPGtEx 25rhNltEiK/Myv//IjSvgjiApgvCSTNLcgqJUtj86OMiSPmATadT+uVKfEk8wG/G yv2bm11GRQxOX79EV4skfIdba6X2/Oiekb5ktmF0GL8iav3tYUVOHbRLYBmJExT1 YMihip/CFuvdzuoUCbFGGlcni9IJAArmADyJEdVmEUEV258eMVtPYiGFEdi8ajM9 H5X+f5SVRuW77lKg+PhSzgjaRRvJUfLeDE0CECHS0sLpWT5EZF+KjhBq5g6a7rIq 4iE4QE7TMPjGtL8tzqXEPJhqb9NMG+29BI+V+SYcT/aJBky2pOTC65rBm6kXT8g= =wTeg -----END PGP SIGNATURE----- _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk<mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake