Hello firedrake, I'm having problems running a firedrake test script across multiple nodes. Petsc crashes with seg faults when using more than one node but the script runs fine when restricted to a single node. The cluster has 4 nodes where each node has 16 cores/64GB mem with Ubuntu 16.04.2 LTS and IB interconnects. We use SLURM but the errors occur using straight mpirun and a hostfile. I've run openmpi and openib tests and they indicate no problems with either subsystem. Firedrake installs cleanly. pturner@ubuntu-0-0:~/mpitest$ which python /home/pturner/firedrake/bin/python
print firedrake.__version__ 0.13.0+1303.g9070020
Test simple firedrake script using mpirun pturner@ubuntu-0-0:~/mpitest$ cat firedrake_proj.py from firedrake import * mesh = UnitSquareMesh(10, 10) p1 = FunctionSpace(mesh, 'CG', 1) f = Function(p1, name='function') x, y = SpatialCoordinate(mesh) expr = sin(2*pi*x)*(1 + y) f.project(expr) n = norm(f) if mesh.comm.rank == 0: print('Norm {:}'.format(n)) print('SUCCESS') Execute with 16 cores (single node): pturner@ubuntu-0-0:~/mpitest$ mpirun --mca btl openib,sm,self --mca mpi_warn_on_fork 0 -n 16 -hostfile ~/hostfile python firedrake_proj.py Norm 1.07999261002 SUCCESS Execute with 32 cores (2 nodes): pturner@ubuntu-0-0:~/mpitest$ mpirun --mca btl openib,sm,self --mca mpi_warn_on_fork 0 -n 32 -hostfile ~/hostfile python firedrake_proj.py Consistent failures on ranks 4 and 20: [20]PETSC ERROR: ------------------------------------------------------------------------ [20]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [...] [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [...] Any idea what might be going wrong? Thx, --Paul Paul J Turner OHSU/CMOP