Hi Lawrence and Miklos, thanks for your replies.
This coarse mesh has 20 cells, so four processes (at least) will not own any cells on the coarse mesh. While most of firedrake and pyop2 will work in this situation, various bits of mesh setup won't. Note that even if the coarsest mesh went through, when you refine 4 of the processes won't have any work.
oh, yes, that's true. It also means that this will limit the strong scaling I can do with my code if I always want at least once cell per processor. I just repeated a run with 20*4 cells on the coarsest level on 24 cores. I use 64 layers and refine the coarse grid 3 times, to 5120*64 327680=3.3E5 cells in total on the finest grid, so that's far away from 2E9 where we run out of integers for global indices. It crashes with a segfault, see below. Eike Number of cells on finest grid = 5120 dx = 364.458 km, dt = 2429.717 s _pmiu_daemon(SIGCHLD): [NID 01160] [c6-0c0s2n0] [Thu Feb 5 14:22:05 2015] PE RANK 11 exit signal Segmentation fault [NID 01160] 2015-02-05 14:22:05 Apid 12880356: initiated application termination Application 12880356 exit codes: 139 Application 12880356 resources: utime ~31s, stime ~19s, Rss ~318352, inblocks ~104428, outblocks ~788 Finished atThu Feb 5 14:22:10 GMT 2015