February 2015 - firedrake - imperial.ac.uk

Re: [firedrake] Crash when running at higher order on ARCHER: resolved now
by Fabio Luporini 14 Feb '15

14 Feb '15

Hi Eike, Sorry for this. I though I had fixed that sort of bug, but probably this is triggered by something else. Actually, it should not even be the wrapper's responsibility to construct arrays of that size - that is, everything should happen in COFFEE. I'm planning to do this in the next few days. In the meanwhile, could you file a bug under "issues" on github/coffee ? could you paste the form, or even better, a sort of minimal test that leads to this error? Thank you -- Fabio 2015-02-14 10:53 GMT+00:00 Eike Mueller <e.mueller(a)bath.ac.uk>: > Dear firedrakers, > > I finally got to the bottom of this. It turns out that I had set > parameters[“COFFEE”][“O2”]= False around the parloop which executes the > kernel, but not around the bit of code which actually compiles the UFL > form. Stupid mistake… So this caused a horrible segfault, since the kernel > expected data of size A[8][20], but it was passed A[6][18]. It was quite > tricky to find this kind of bug, though, since it only segfaults without > much information. I finally managed to run interactively on ARCHER, > inspected the core dump with gdb and looked at the generated c-code. I was > wondering whether this kind of issue can be detected when you generate the > wrapper code? Don’t you know both the signature of the function and the > passed data at this point? > > Or has the COFFEE optimisation issue been resolved? I pulled the latest > version of COFFEE, though. > > Thanks, > > Eike > > -- > > Dr Eike Hermann Mueller > Research Associate (PostDoc) > > Department of Mathematical Sciences > University of Bath > Bath BA2 7AY, United Kingdom > > +44 1225 38 5803 > e.mueller(a)bath.ac.uk > http://people.bath.ac.uk/em459/ > > On 5 Feb 2015, at 16:28, Eike Mueller <E.Mueller(a)bath.ac.uk> wrote: > > Thanks, I tried the atp and also inspected the core dump with > > gdb python core > > There is no backtrace in the core dump, and ATP does not generate any > information either. > > I still only get the segfault in my output file. I hope I can localise > this a bit more tomorrow. > > Eike > > On 05/02/15 15:23, Patrick Farrell wrote: > > On 05/02/15 14:37, Lawrence Mitchell wrote: > > > Number of cells on finest grid = 5120 dx = 364.458 km, dt = > 2429.717 s _pmiu_daemon(SIGCHLD): [NID 01160] [c6-0c0s2n0] [Thu Feb > 5 14:22:05 2015] PE RANK 11 exit signal Segmentation fault [NID > 01160] 2015-02-05 14:22:05 Apid 12880356: initiated application > termination Application 12880356 exit codes: 139 Application > 12880356 resources: utime ~31s, stime ~19s, Rss ~318352, inblocks > ~104428, outblocks ~788 Finished atThu Feb 5 14:22:10 GMT 2015 > > > Hmm, that's not a lot of useful information. > > > Try running again with > > module load atp > export ATP_ENABLED=1 > > Sometimes it gives useful information about abnormal terminations; > http://www.archer.ac.uk/documentation/best-practice-guide/debug.php > > Patrick > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > >

2 1

Re: [firedrake] clapack.h
by David Ham 10 Feb '15

10 Feb '15

On 9 February 2015 at 18:15, Mitchell, Lawrence < lawrence.mitchell(a)imperial.ac.uk> wrote: > > On 9 Feb 2015, at 18:11, Colin Cotter <colin.cotter(a)imperial.ac.uk> wrote: > > > Dear Firedrakers, > > David has been helping me by implementing a way of assembling the > inverse of a local tensor. This involves calling LAPACK routines, and we > realised that the big problem is how to locate clapack, and how to link > against it. > > > > The only approach we could think of was that scipy.linalg.lapack > builds a clapack.so. Then, we could make our own header file that contains > the interfaces to the LAPACK routines that we want to use, and compile > aginst clapack.so. We really don't want to replicate some kind of CMAKE > approach to locating libraries for COFFEE. > > Use Eigen, which is header only? > I don't understand why that helps - now we have to find Eigen (finding headers versus finding libraries would not seem to be a huge distinction). > > Lawrence > -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

2 1

Re: [firedrake] clapack.h
by Fabio Luporini 10 Feb '15

10 Feb '15

Just *out of curiosity*: I *think* part of the machinery for supporting Eigen is already in place. Exporting the env var "PYOP2_BLAS_DIR" to the Eigen's main folder should do the trick (it's not just finding the headers because Eigen is c++, and you need to wrap your kernel's signature with extern "C" for linking to work), provided that one makes the condition "if self._kernel._applied_blas:" - line 655 in host.py - more general. 2015-02-10 10:03 GMT+00:00 Ham, David A <david.ham(a)imperial.ac.uk>: > > > On 9 February 2015 at 18:15, Mitchell, Lawrence < > lawrence.mitchell(a)imperial.ac.uk> wrote: > >> >> On 9 Feb 2015, at 18:11, Colin Cotter <colin.cotter(a)imperial.ac.uk> >> wrote: >> >> > Dear Firedrakers, >> > David has been helping me by implementing a way of assembling the >> inverse of a local tensor. This involves calling LAPACK routines, and we >> realised that the big problem is how to locate clapack, and how to link >> against it. >> > >> > The only approach we could think of was that scipy.linalg.lapack >> builds a clapack.so. Then, we could make our own header file that contains >> the interfaces to the LAPACK routines that we want to use, and compile >> aginst clapack.so. We really don't want to replicate some kind of CMAKE >> approach to locating libraries for COFFEE. >> >> Use Eigen, which is header only? >> > > I don't understand why that helps - now we have to find Eigen (finding > headers versus finding libraries would not seem to be a huge distinction). > > >> >> Lawrence >> > > > > -- > Dr David Ham > Departments of Mathematics and Computing > Imperial College London > > http://www.imperial.ac.uk/people/david.ham >

1 0

Re: [firedrake] clapack.h
by Fabio Luporini 09 Feb '15

09 Feb '15

Just a quick note about performance: since LAPACK is heavily based on BLAS level 3, and since BLAS level 3 are *usually* tuned for large matrices, bear in mind that using it repeatedly for operations on "small" objects (i.e., that fit some levels of cache) might actually slow down the computation. Eigen could also have the advantage of working around this issue, since it tends to perform better for small size problems than the various BLAS (and BLAS-like) alternatives. This (various BLAS implementations vs Eigen vs pure C) is something I started investigating with/through COFFEE, but that I eventually dropped for lack of time I don't actually think this info could be of any help, but you know, just in case -- Fabio

1 0

clapack.h
by Colin Cotter 09 Feb '15

09 Feb '15

Dear Firedrakers, David has been helping me by implementing a way of assembling the inverse of a local tensor. This involves calling LAPACK routines, and we realised that the big problem is how to locate clapack, and how to link against it. The only approach we could think of was that scipy.linalg.lapack builds a clapack.so. Then, we could make our own header file that contains the interfaces to the LAPACK routines that we want to use, and compile aginst clapack.so. We really don't want to replicate some kind of CMAKE approach to locating libraries for COFFEE. What is the best way to do this? cheers --cjc

3 2

Re: [firedrake] Comparison PETSc and matrix-free
by Eike Mueller 03 Feb '15

03 Feb '15

Hi Lawrence, > Below are the first weak scaling results from runs at lowest order on up > to 96 cores on ARCHER. On 384 cores the code crashes with a PETSc error > (segfault). This crash is already in the matrix-free solver (which, of > course, uses a PETSc KSP). Could this be an issue with the python module for launching the compilation/loading the kernels in PyOP2? However, on Friday I ran with PYOP2_NO_FORK_AVAILABLE=1, which I thought would fix this? If I run with PYOP2_NO_FORK_AVAILABLE=0, then it crashes with a different error because it can't compile a kernel. This morning just repeated exactly the same 384 core run (PYOP2_NO_FORK_AVAILABLE=1 as before) and now it goes through without problems (i.e. it does both the matrix-free and the PETSc solve). I observe something similar with the 1536 run: The first run crashed, in the subsequent runs the matrix-free solver completes but it crashes later in the run where it gets to the PETSc solve. I then set PYOP2_DEBUG=1 in the 1536 core run, and again it fails because it can't compile code. The resulting .err file is empty. I then ran the compilation command in the .log file manually. It goes through, but with a warning, which I attach together with the output of the run. Cheers, Eike

2 3