Dear firedrakers, I still get a crash when running at higher order on ARCHER (but it runs fine at lowest order). The output is below. This is odd, since the problem appears to be in the mesh generation, which should be the same for lowest- and higher order? This time I chose the problem size small enough (problems of this size run on my laptop, and I use a full node on ARCHER). I use the knepley/fix-plex-1d-refinement branch of PETSc and the firedrake branch of petsc4py On my laptop (where it works) I use dmplex-1d-refinement branch of petsc. Thanks, Eike eike@eslogin002 $ cat helmholtz_2692059.sdb/output.log Running helmholtz PBS_JOBID = 2692059.sdb Started atThu Feb 5 08:39:45 GMT 2015 +---------------------------+ ! Mixed Gravity wave solver ! +---------------------------+ Running on 24 MPI processes *** Parameters *** General: solve_matrixfree = True solve_petsc = True warmup_run = True orography = False nu_cfl = 10.0 speed_N = 0.01 speed_c = 300.0 Output: savetodisk = False output_dir = output Grid: nlayer = 64 ref_count_coarse = 0 nlevel = 3 thickness = 10000.0 Mixed system: higher_order = True verbose = 2 schur_diagonal_only = False maxiter = 20 tolerance = 1e-05 ksp_type = gmres Pressure solve: maxiter = 3 tolerance = 1e-14 verbose = 1 ksp_type = cg Multigrid: mu_relax = 0.8 n_postsmooth = 1 n_coarsesmooth = 1 n_presmooth = 1 Traceback (most recent call last): File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 486, in <module> Traceback (most recent call last): File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 486, in <module> Traceback (most recent call last): File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 486, in <module> Traceback (most recent call last): File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 486, in <module> main(parameter_filename) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 256, in main main(parameter_filename) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 256, in main param_grid['thickness']) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 146, in build_mesh_hierarchy param_grid['thickness']) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 146, in build_mesh_hierarchy refinement_level=ref_count_coarse) refinement_level=ref_count_coarse) main(parameter_filename) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line 560, in IcosahedralSphereMesh File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line 560, in IcosahedralSphereMesh File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 256, in main param_grid['thickness']) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 146, in build_mesh_hierarchy main(parameter_filename) refinement_level=ref_count_coarse) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 256, in main File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line 560, in IcosahedralSphereMesh m = mesh.Mesh(plex, dim=3, reorder=reorder) m = mesh.Mesh(plex, dim=3, reorder=reorder) File "<string>", line 2, in Mesh File "<string>", line 2, in Mesh m = mesh.Mesh(plex, dim=3, reorder=reorder) File "<string>", line 2, in Mesh param_grid['thickness']) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 146, in build_mesh_hierarchy refinement_level=ref_count_coarse) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line 560, in IcosahedralSphereMesh m = mesh.Mesh(plex, dim=3, reorder=reorder) File "<string>", line 2, in Mesh File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py", line 197, in wrapper File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py", line 197, in wrapper File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py", line 197, in wrapper File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py", line 197, in wrapper return f(*args, **kwargs) return f(*args, **kwargs) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py", line 210, in Mesh File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py", line 210, in Mesh return f(*args, **kwargs) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py", line 210, in Mesh return f(*args, **kwargs) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py", line 210, in Mesh cell_facets = plex.getConeSize(cStart) cell_facets = plex.getConeSize(cStart) cell_facets = plex.getConeSize(cStart) cell_facets = plex.getConeSize(cStart) File "PETSc/DMPlex.pyx", line 140, in petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913) File "PETSc/DMPlex.pyx", line 140, in petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913) File "PETSc/DMPlex.pyx", line 140, in petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913) File "PETSc/DMPlex.pyx", line 140, in petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913) AssertionError AssertionError AssertionError AssertionError Rank 13 [Thu Feb 5 08:40:23 2015] [c7-0c2s11n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 13 Rank 19 [Thu Feb 5 08:40:23 2015] [c7-0c2s11n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 19 Rank 1 [Thu Feb 5 08:40:23 2015] [c7-0c2s11n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 Rank 7 [Thu Feb 5 08:40:23 2015] [c7-0c2s11n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7 _pmiu_daemon(SIGCHLD): [NID 01517] [c7-0c2s11n1] [Thu Feb 5 08:40:23 2015] PE RANK 1 exit signal Aborted [NID 01517] 2015-02-05 08:40:23 Apid 12877131: initiated application termination Application 12877131 exit codes: 134 Application 12877131 resources: utime ~1s, stime ~15s, Rss ~59684, inblocks ~104518, outblocks ~790 Finished atThu Feb 5 08:40:25 GMT 2015