Re: [firedrake] Crash when running at higher order on ARCHER

5 Feb 2015

      Hello Eike,

It fails exactly before the Mesh factory function would identify whether to create a simplex or quadrilateral mesh object. It tries to retrieve how many facets a cell has, which fails the assertion pStart <= cStart < pEnd. The only explanation I have is that some MPI processes own 0 (zero) cells of the mesh.

Maybe try using less MPI nodes or a higher refinement (more cells).

Regards,
Miklos

________________________________________
From: firedrake-bounces@imperial.ac.uk [firedrake-bounces@imperial.ac.uk] on behalf of Eike Mueller [E.Mueller@bath.ac.uk]
Sent: 05 February 2015 08:57
To: firedrake
Subject: [firedrake] Crash when running at higher order on ARCHER

Dear firedrakers,

I still get a crash when running at higher order on ARCHER (but it runs
fine at lowest order). The output is below. This is odd, since the
problem appears to be in the mesh generation, which should be the same
for lowest- and higher order?

This time I chose the problem size small enough (problems of this size
run on my laptop, and I use a full node on ARCHER).

I use the knepley/fix-plex-1d-refinement branch of PETSc and the
firedrake branch of petsc4py

On my laptop (where it works) I use dmplex-1d-refinement branch of petsc.

Thanks,

Eike

eike@eslogin002 $ cat helmholtz_2692059.sdb/output.log

Running helmholtz

PBS_JOBID = 2692059.sdb

Started atThu Feb  5 08:39:45 GMT 2015
+---------------------------+
! Mixed Gravity wave solver !
+---------------------------+

Running on       24 MPI processes

*** Parameters ***
General:
      solve_matrixfree = True
      solve_petsc = True
      warmup_run = True
      orography = False
      nu_cfl = 10.0
      speed_N = 0.01
      speed_c = 300.0

Output:
      savetodisk = False
      output_dir = output

Grid:
      nlayer = 64
      ref_count_coarse = 0
      nlevel = 3
      thickness = 10000.0

Mixed system:
      higher_order = True
      verbose = 2
      schur_diagonal_only = False
      maxiter = 20
      tolerance = 1e-05
      ksp_type = gmres

Pressure solve:
      maxiter = 3
      tolerance = 1e-14
      verbose = 1
      ksp_type = cg

Multigrid:
      mu_relax = 0.8
      n_postsmooth = 1
      n_coarsesmooth = 1
      n_presmooth = 1

Traceback (most recent call last):
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 486, in <module>
Traceback (most recent call last):
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 486, in <module>
Traceback (most recent call last):
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 486, in <module>
Traceback (most recent call last):
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 486, in <module>
     main(parameter_filename)
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 256, in main
     main(parameter_filename)
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 256, in main
     param_grid['thickness'])
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 146, in build_mesh_hierarchy
     param_grid['thickness'])
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 146, in build_mesh_hierarchy
     refinement_level=ref_count_coarse)
     refinement_level=ref_count_coarse)
     main(parameter_filename)
   File
"/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line
560, in IcosahedralSphereMesh
   File
"/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line
560, in IcosahedralSphereMesh
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 256, in main
     param_grid['thickness'])
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 146, in build_mesh_hierarchy
     main(parameter_filename)
     refinement_level=ref_count_coarse)
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 256, in main
   File
"/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line
560, in IcosahedralSphereMesh
     m = mesh.Mesh(plex, dim=3, reorder=reorder)
     m = mesh.Mesh(plex, dim=3, reorder=reorder)
   File "<string>", line 2, in Mesh
   File "<string>", line 2, in Mesh
     m = mesh.Mesh(plex, dim=3, reorder=reorder)
   File "<string>", line 2, in Mesh
     param_grid['thickness'])
   File
"/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py",
line 146, in build_mesh_hierarchy
     refinement_level=ref_count_coarse)
   File
"/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line
560, in IcosahedralSphereMesh
     m = mesh.Mesh(plex, dim=3, reorder=reorder)
   File "<string>", line 2, in Mesh
   File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py",
line 197, in wrapper
   File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py",
line 197, in wrapper
   File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py",
line 197, in wrapper
   File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py",
line 197, in wrapper
     return f(*args, **kwargs)
     return f(*args, **kwargs)
   File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py",
line 210, in Mesh
   File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py",
line 210, in Mesh
     return f(*args, **kwargs)
   File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py",
line 210, in Mesh
     return f(*args, **kwargs)
   File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py",
line 210, in Mesh
         cell_facets = plex.getConeSize(cStart)
cell_facets = plex.getConeSize(cStart)
         cell_facets = plex.getConeSize(cStart)
cell_facets = plex.getConeSize(cStart)
   File "PETSc/DMPlex.pyx", line 140, in
petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913)
   File "PETSc/DMPlex.pyx", line 140, in
petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913)
   File "PETSc/DMPlex.pyx", line 140, in
petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913)
   File "PETSc/DMPlex.pyx", line 140, in
petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913)
AssertionError
AssertionError
AssertionError
AssertionError
Rank 13 [Thu Feb  5 08:40:23 2015] [c7-0c2s11n1] application called
MPI_Abort(MPI_COMM_WORLD, 1) - process 13
Rank 19 [Thu Feb  5 08:40:23 2015] [c7-0c2s11n1] application called
MPI_Abort(MPI_COMM_WORLD, 1) - process 19
Rank 1 [Thu Feb  5 08:40:23 2015] [c7-0c2s11n1] application called
MPI_Abort(MPI_COMM_WORLD, 1) - process 1
Rank 7 [Thu Feb  5 08:40:23 2015] [c7-0c2s11n1] application called
MPI_Abort(MPI_COMM_WORLD, 1) - process 7
_pmiu_daemon(SIGCHLD): [NID 01517] [c7-0c2s11n1] [Thu Feb  5 08:40:23
2015] PE RANK 1 exit signal Aborted
[NID 01517] 2015-02-05 08:40:23 Apid 12877131: initiated application
termination
Application 12877131 exit codes: 134
Application 12877131 resources: utime ~1s, stime ~15s, Rss ~59684,
inblocks ~104518, outblocks ~790
Finished atThu Feb  5 08:40:25 GMT 2015

_______________________________________________
firedrake mailing list
firedrake@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/firedrake

Re: [firedrake] Crash when running at higher order on ARCHER

Homolya, Miklós