- firedrake - imperial.ac.uk

Re: [firedrake] Problems running on more than one node (ARCHER)
by Andrew McRae 16 Feb '15

16 Feb '15

Ah, my .env file was missing some magic lines. PYOP2_CACHE_DIR=$WORK/pyop2-cache export PYOP2_CACHE_DIR FIREDRAKE_FFC_KERNEL_CACHE_DIR=$WORK/firedrake-ffc-cache export FIREDRAKE_FFC_KERNEL_CACHE_DIR On 15 February 2015 at 23:27, Rathgeber, Florian < florian.rathgeber(a)imperial.ac.uk> wrote: > Don't use /tmp on ARCHER! It's mounted to ramfs, so you *really* > shouldn't write anything there. See also > > http://archer.ac.uk/documentation/user-guide/resource_management.php#sec-3.… > > On 2/15/2015 11:01 PM, Miklos Homolya wrote: > > Uh, is /tmp globally visible on ARCHER, or is it local to each node? > > Is it just rank 0, who writes the generated kernel to file? > > > > If the answers are local and /tmp, and rank 0 writing the kernel, then > it’s easy to see why it fails. > > > >> On 15 Feb 2015, at 22:42, Andrew McRae <a.mcrae12(a)imperial.ac.uk> > wrote: > >> > >> I have a script which runs fine on 1 node (24 cores) but fails on 2 or > more. > >> > >> It seems to die somewhere during the extruded mesh generation, with > messages such as > >> > >> OSError: /tmp/pyop2-cache-uid15427/9d8ebf65631aa626aeb3cd8098e2662f.so: > cannot open shared object file: No such file or directory > >> > >> Full output and error files attached. > >

1 0

Crash when running at higher order on ARCHER
by Eike Mueller 16 Feb '15

16 Feb '15

Dear firedrakers, I still get a crash when running at higher order on ARCHER (but it runs fine at lowest order). The output is below. This is odd, since the problem appears to be in the mesh generation, which should be the same for lowest- and higher order? This time I chose the problem size small enough (problems of this size run on my laptop, and I use a full node on ARCHER). I use the knepley/fix-plex-1d-refinement branch of PETSc and the firedrake branch of petsc4py On my laptop (where it works) I use dmplex-1d-refinement branch of petsc. Thanks, Eike eike@eslogin002 $ cat helmholtz_2692059.sdb/output.log Running helmholtz PBS_JOBID = 2692059.sdb Started atThu Feb 5 08:39:45 GMT 2015 +---------------------------+ ! Mixed Gravity wave solver ! +---------------------------+ Running on 24 MPI processes *** Parameters *** General: solve_matrixfree = True solve_petsc = True warmup_run = True orography = False nu_cfl = 10.0 speed_N = 0.01 speed_c = 300.0 Output: savetodisk = False output_dir = output Grid: nlayer = 64 ref_count_coarse = 0 nlevel = 3 thickness = 10000.0 Mixed system: higher_order = True verbose = 2 schur_diagonal_only = False maxiter = 20 tolerance = 1e-05 ksp_type = gmres Pressure solve: maxiter = 3 tolerance = 1e-14 verbose = 1 ksp_type = cg Multigrid: mu_relax = 0.8 n_postsmooth = 1 n_coarsesmooth = 1 n_presmooth = 1 Traceback (most recent call last): File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 486, in <module> Traceback (most recent call last): File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 486, in <module> Traceback (most recent call last): File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 486, in <module> Traceback (most recent call last): File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 486, in <module> main(parameter_filename) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 256, in main main(parameter_filename) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 256, in main param_grid['thickness']) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 146, in build_mesh_hierarchy param_grid['thickness']) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 146, in build_mesh_hierarchy refinement_level=ref_count_coarse) refinement_level=ref_count_coarse) main(parameter_filename) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line 560, in IcosahedralSphereMesh File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line 560, in IcosahedralSphereMesh File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 256, in main param_grid['thickness']) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 146, in build_mesh_hierarchy main(parameter_filename) refinement_level=ref_count_coarse) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 256, in main File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line 560, in IcosahedralSphereMesh m = mesh.Mesh(plex, dim=3, reorder=reorder) m = mesh.Mesh(plex, dim=3, reorder=reorder) File "<string>", line 2, in Mesh File "<string>", line 2, in Mesh m = mesh.Mesh(plex, dim=3, reorder=reorder) File "<string>", line 2, in Mesh param_grid['thickness']) File "/work/n02/n02/eike//git_workspace/firedrake-helmholtzsolver/source/driver.py", line 146, in build_mesh_hierarchy refinement_level=ref_count_coarse) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/utility_meshes.py", line 560, in IcosahedralSphereMesh m = mesh.Mesh(plex, dim=3, reorder=reorder) File "<string>", line 2, in Mesh File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py", line 197, in wrapper File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py", line 197, in wrapper File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py", line 197, in wrapper File "/work/n02/n02/eike/git_workspace/PyOP2/pyop2/profiling.py", line 197, in wrapper return f(*args, **kwargs) return f(*args, **kwargs) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py", line 210, in Mesh File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py", line 210, in Mesh return f(*args, **kwargs) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py", line 210, in Mesh return f(*args, **kwargs) File "/work/n02/n02/eike/git_workspace/firedrake/firedrake/mesh.py", line 210, in Mesh cell_facets = plex.getConeSize(cStart) cell_facets = plex.getConeSize(cStart) cell_facets = plex.getConeSize(cStart) cell_facets = plex.getConeSize(cStart) File "PETSc/DMPlex.pyx", line 140, in petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913) File "PETSc/DMPlex.pyx", line 140, in petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913) File "PETSc/DMPlex.pyx", line 140, in petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913) File "PETSc/DMPlex.pyx", line 140, in petsc4py.PETSc.DMPlex.getConeSize (src/petsc4py.PETSc.c:203913) AssertionError AssertionError AssertionError AssertionError Rank 13 [Thu Feb 5 08:40:23 2015] [c7-0c2s11n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 13 Rank 19 [Thu Feb 5 08:40:23 2015] [c7-0c2s11n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 19 Rank 1 [Thu Feb 5 08:40:23 2015] [c7-0c2s11n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 Rank 7 [Thu Feb 5 08:40:23 2015] [c7-0c2s11n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7 _pmiu_daemon(SIGCHLD): [NID 01517] [c7-0c2s11n1] [Thu Feb 5 08:40:23 2015] PE RANK 1 exit signal Aborted [NID 01517] 2015-02-05 08:40:23 Apid 12877131: initiated application termination Application 12877131 exit codes: 134 Application 12877131 resources: utime ~1s, stime ~15s, Rss ~59684, inblocks ~104518, outblocks ~790 Finished atThu Feb 5 08:40:25 GMT 2015

5 8

Problems running on more than one node (ARCHER)
by Andrew McRae 15 Feb '15

15 Feb '15

I have a script which runs fine on 1 node (24 cores) but fails on 2 or more. It seems to die somewhere during the extruded mesh generation, with messages such as OSError: /tmp/pyop2-cache-uid15427/9d8ebf65631aa626aeb3cd8098e2662f.so: cannot open shared object file: No such file or directory Full output and error files attached.

3 2

Re: [firedrake] Crash when running at higher order on ARCHER: resolved now
by Fabio Luporini 14 Feb '15

14 Feb '15

Hi Eike, Sorry for this. I though I had fixed that sort of bug, but probably this is triggered by something else. Actually, it should not even be the wrapper's responsibility to construct arrays of that size - that is, everything should happen in COFFEE. I'm planning to do this in the next few days. In the meanwhile, could you file a bug under "issues" on github/coffee ? could you paste the form, or even better, a sort of minimal test that leads to this error? Thank you -- Fabio 2015-02-14 10:53 GMT+00:00 Eike Mueller <e.mueller(a)bath.ac.uk>: > Dear firedrakers, > > I finally got to the bottom of this. It turns out that I had set > parameters[“COFFEE”][“O2”]= False around the parloop which executes the > kernel, but not around the bit of code which actually compiles the UFL > form. Stupid mistake… So this caused a horrible segfault, since the kernel > expected data of size A[8][20], but it was passed A[6][18]. It was quite > tricky to find this kind of bug, though, since it only segfaults without > much information. I finally managed to run interactively on ARCHER, > inspected the core dump with gdb and looked at the generated c-code. I was > wondering whether this kind of issue can be detected when you generate the > wrapper code? Don’t you know both the signature of the function and the > passed data at this point? > > Or has the COFFEE optimisation issue been resolved? I pulled the latest > version of COFFEE, though. > > Thanks, > > Eike > > -- > > Dr Eike Hermann Mueller > Research Associate (PostDoc) > > Department of Mathematical Sciences > University of Bath > Bath BA2 7AY, United Kingdom > > +44 1225 38 5803 > e.mueller(a)bath.ac.uk > http://people.bath.ac.uk/em459/ > > On 5 Feb 2015, at 16:28, Eike Mueller <E.Mueller(a)bath.ac.uk> wrote: > > Thanks, I tried the atp and also inspected the core dump with > > gdb python core > > There is no backtrace in the core dump, and ATP does not generate any > information either. > > I still only get the segfault in my output file. I hope I can localise > this a bit more tomorrow. > > Eike > > On 05/02/15 15:23, Patrick Farrell wrote: > > On 05/02/15 14:37, Lawrence Mitchell wrote: > > > Number of cells on finest grid = 5120 dx = 364.458 km, dt = > 2429.717 s _pmiu_daemon(SIGCHLD): [NID 01160] [c6-0c0s2n0] [Thu Feb > 5 14:22:05 2015] PE RANK 11 exit signal Segmentation fault [NID > 01160] 2015-02-05 14:22:05 Apid 12880356: initiated application > termination Application 12880356 exit codes: 139 Application > 12880356 resources: utime ~31s, stime ~19s, Rss ~318352, inblocks > ~104428, outblocks ~788 Finished atThu Feb 5 14:22:10 GMT 2015 > > > Hmm, that's not a lot of useful information. > > > Try running again with > > module load atp > export ATP_ENABLED=1 > > Sometimes it gives useful information about abnormal terminations; > http://www.archer.ac.uk/documentation/best-practice-guide/debug.php > > Patrick > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > >

2 1

Re: [firedrake] clapack.h
by David Ham 10 Feb '15

10 Feb '15

On 9 February 2015 at 18:15, Mitchell, Lawrence < lawrence.mitchell(a)imperial.ac.uk> wrote: > > On 9 Feb 2015, at 18:11, Colin Cotter <colin.cotter(a)imperial.ac.uk> wrote: > > > Dear Firedrakers, > > David has been helping me by implementing a way of assembling the > inverse of a local tensor. This involves calling LAPACK routines, and we > realised that the big problem is how to locate clapack, and how to link > against it. > > > > The only approach we could think of was that scipy.linalg.lapack > builds a clapack.so. Then, we could make our own header file that contains > the interfaces to the LAPACK routines that we want to use, and compile > aginst clapack.so. We really don't want to replicate some kind of CMAKE > approach to locating libraries for COFFEE. > > Use Eigen, which is header only? > I don't understand why that helps - now we have to find Eigen (finding headers versus finding libraries would not seem to be a huge distinction). > > Lawrence > -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

2 1

Re: [firedrake] clapack.h
by Fabio Luporini 10 Feb '15

10 Feb '15

Just *out of curiosity*: I *think* part of the machinery for supporting Eigen is already in place. Exporting the env var "PYOP2_BLAS_DIR" to the Eigen's main folder should do the trick (it's not just finding the headers because Eigen is c++, and you need to wrap your kernel's signature with extern "C" for linking to work), provided that one makes the condition "if self._kernel._applied_blas:" - line 655 in host.py - more general. 2015-02-10 10:03 GMT+00:00 Ham, David A <david.ham(a)imperial.ac.uk>: > > > On 9 February 2015 at 18:15, Mitchell, Lawrence < > lawrence.mitchell(a)imperial.ac.uk> wrote: > >> >> On 9 Feb 2015, at 18:11, Colin Cotter <colin.cotter(a)imperial.ac.uk> >> wrote: >> >> > Dear Firedrakers, >> > David has been helping me by implementing a way of assembling the >> inverse of a local tensor. This involves calling LAPACK routines, and we >> realised that the big problem is how to locate clapack, and how to link >> against it. >> > >> > The only approach we could think of was that scipy.linalg.lapack >> builds a clapack.so. Then, we could make our own header file that contains >> the interfaces to the LAPACK routines that we want to use, and compile >> aginst clapack.so. We really don't want to replicate some kind of CMAKE >> approach to locating libraries for COFFEE. >> >> Use Eigen, which is header only? >> > > I don't understand why that helps - now we have to find Eigen (finding > headers versus finding libraries would not seem to be a huge distinction). > > >> >> Lawrence >> > > > > -- > Dr David Ham > Departments of Mathematics and Computing > Imperial College London > > http://www.imperial.ac.uk/people/david.ham >

1 0

Re: [firedrake] clapack.h
by Fabio Luporini 09 Feb '15

09 Feb '15

Just a quick note about performance: since LAPACK is heavily based on BLAS level 3, and since BLAS level 3 are *usually* tuned for large matrices, bear in mind that using it repeatedly for operations on "small" objects (i.e., that fit some levels of cache) might actually slow down the computation. Eigen could also have the advantage of working around this issue, since it tends to perform better for small size problems than the various BLAS (and BLAS-like) alternatives. This (various BLAS implementations vs Eigen vs pure C) is something I started investigating with/through COFFEE, but that I eventually dropped for lack of time I don't actually think this info could be of any help, but you know, just in case -- Fabio

1 0

clapack.h
by Colin Cotter 09 Feb '15

09 Feb '15

Dear Firedrakers, David has been helping me by implementing a way of assembling the inverse of a local tensor. This involves calling LAPACK routines, and we realised that the big problem is how to locate clapack, and how to link against it. The only approach we could think of was that scipy.linalg.lapack builds a clapack.so. Then, we could make our own header file that contains the interfaces to the LAPACK routines that we want to use, and compile aginst clapack.so. We really don't want to replicate some kind of CMAKE approach to locating libraries for COFFEE. What is the best way to do this? cheers --cjc

3 2

Re: [firedrake] Comparison PETSc and matrix-free
by Eike Mueller 03 Feb '15

03 Feb '15

Hi Lawrence, > Below are the first weak scaling results from runs at lowest order on up > to 96 cores on ARCHER. On 384 cores the code crashes with a PETSc error > (segfault). This crash is already in the matrix-free solver (which, of > course, uses a PETSc KSP). Could this be an issue with the python module for launching the compilation/loading the kernels in PyOP2? However, on Friday I ran with PYOP2_NO_FORK_AVAILABLE=1, which I thought would fix this? If I run with PYOP2_NO_FORK_AVAILABLE=0, then it crashes with a different error because it can't compile a kernel. This morning just repeated exactly the same 384 core run (PYOP2_NO_FORK_AVAILABLE=1 as before) and now it goes through without problems (i.e. it does both the matrix-free and the PETSc solve). I observe something similar with the 1536 run: The first run crashed, in the subsequent runs the matrix-free solver completes but it crashes later in the run where it gets to the PETSc solve. I then set PYOP2_DEBUG=1 in the 1536 core run, and again it fails because it can't compile code. The resulting .err file is empty. I then ran the compilation command in the .log file manually. It goes through, but with a warning, which I attach together with the output of the run. Cheers, Eike

2 3

FInAT mailing list
by David Ham 31 Jan '15

31 Jan '15

Dear Firedrakers, Since FInAT involves people outside the Firedrake project who have no interest in our chatter, I have established a FInAT mailing list. Any of you who wish to follow FInAT development discussions are invited to subscribe to the list: https://mailman.ic.ac.uk/mailman/listinfo/finat Regards, David -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

1 0