- firedrake - imperial.ac.uk

Re: [firedrake] Valgrind and Firedrake
by Lawrence Mitchell 09 Jun '16

09 Jun '16

Please send emails about firedrake issues to the firedrake list, or submit github issues. The intersection of the expertise is larger than that of any one individual, moreover you're much more likely to get an answer quickly. > On 9 Jun 2016, at 14:51, Nicolas Barral <n.barral(a)imperial.ac.uk> wrote: > > Hi Lawrence, > > I am still trying to find the bug I mentioned on IRC two weeks ago (point evaluation raising a not in domain error for points in the domain), and I have been totally unsuccessful so I need your help. > > You suggested using valgrind to find a memory issue, which I tried with the regular build of Firedrake (not in debug mode). Even with the suppression file suggested on Python's website, I get a many thousand line log, most errors being unusable because the debug symbols were not exported. Do any of the lines look like they are memory errors in likely candidates. That is, either code compiled by us (library has some kind of md5 hash in its name) or else inside libspatialindex? > So I thought I should try to compile python with the adequate debug options, and use this python with Firedrake. But how do I do that ? I tried to run firedrake-install with a PATH pointing to my custom build of Python (Miklos's suggestion), but it fails when checking/installing virtualenv (it keeps stopping at "Requirement already satisfied Virtual env installed. Please run firedrake-install again."). It sees the virtualenv from the standard python, which might be a problem ? So I'm stuck at that point. I do not know how to install multiple versions of python and get them to pick up the correct virtualenv and similar. I would have thought that your hand-installed python would not find the globally installed virtualenv stuff, but maybe not. Can you just make a virtualenv "by hand". We effectively just run "python -c 'import virtualenv'" and use that. So if that doesn't work that's somewhere to start. Normally if you're on linux you can just install the package that adds debug symbols for python (on ubuntu this is called python-debug I think). > My plan was then to compile the modules/dependencies in debug mode (notably the libspatialindex). Would that be really helpful ? And how do I do that since I can't just run configure/make in the right directory ? firedrake-install runs configure/make for you. You can determine how to installed the shared libraries in debug mode and add that as an option to firedrake-install, we will gladly accept patches for this. > Another idea came to me, and I searched for the cells of the mesh surrounding the points "not in domain" and computed their barycentric coordinates in the corresponding cell. It turns out that the point is always on a vertex or an edge (one or two barycentric coordinates ~ 1e-16). Could there be a bug in libspatialindex or the at function in these cases ? So what you're saying is that the point is right on the boundary between two cells? I guess there are two ways the point location can fail. Either the point is not found in any bounding box. Or, the point is found by libspatialindex in a bounding box. But that bucket does not actually contain the cell in question, so then the linear search for point location may fail due to floating point rounding. Which of these two are occurring? All of this code is compiled by us and is pretty straight line C, so you can just compile with debugging and step through it in the failing case and figure out what's going on. > I wanted to talk to you about all that in person, but I'm not sure we'll see each other before a while, since I'll be in Bath next week until wednesday... The faulty piece of code is likely to be removed when we will have a good Firedrake implementation of the interpolation and the metric computation, but maybe there's something deeper I'm not seeing which is worth being understood, so any help would be helpful :) I am at Imperial on Thursday next week. Lawrence

3 3

Re: [firedrake] Wrong matrix assembly
by Fabio Luporini 08 Jun '16

08 Jun '16

Hello Anna, under the Inequality constraint folder, I'm running "python buoy-swe.py" and, besides the TSFC/COFFEE prints, I'm obtaining: mu_bar ------ [ 0.34693095 0.28326793 0.20030068 0. -0. ] Matrix b_mu ----------- [[ 1.37331778 0.6536432 0. 0. 0. ] [ 0.6536432 2.33042494 0.50155807 0. 0.] [ 0. 0.50155807 1.54031885 0.20775216 0.] [ 0. 0. 0.20775216 0.20775216 0. ] [ 0. 0. 0. 0. 0. ]] What values do you expect for b_mu ? Am I running the right thing? I tried switching off some optimisations but I still get the same output. Thanks! -- Fabio 2016-06-08 15:16 GMT+02:00 Anna Kalogirou <a.kalogirou(a)leeds.ac.uk>: > Hi Fabio, > > Sorry for that. It should be public now. > > Anna. > > > > On 08/06/16 13:56, Fabio Luporini wrote: > > Hello Anna, > > the code is not accessible as it appears to be in a private repository. > Could you fix this? Thanks > > -- Fabio > > 2016-06-08 14:10 GMT+02:00 Anna Kalogirou <a.kalogirou(a)leeds.ac.uk>: > >> Dear all, >> >> I have a simple question regarding a matrix assembly in Firedrake which I >> believe is not computed correctly. >> >> The relevant code can be found here >> <https://bitbucket.org/annakalog/buoy2d/src/14334d3c20b9f10ed7c1246cde9e3cb6…> >> and the issue is with the B_mu matrix in solvers.py, line 51. I compare the >> result with my own Matlab code and the two don't agree. Also, the rest of >> the matrices M, A, C_lam agree in both sets of code. >> >> Can someone help with what the problem might be? >> >> Thanks, >> >> Anna. >> >> -- >> >> >> Dr Anna Kalogirou >> Research Fellow >> School of Mathematics >> University of Leeds >> >> http://www1.maths.leeds.ac.uk/~matak/ >> >> > > > _______________________________________________ > firedrake mailing listfiredrake@imperial.ac.ukhttps://mailman.ic.ac.uk/mailman/listinfo/firedrake > > >

2 1

Hadamard (element wise) product
by Floriane Gidel [RPG] 08 Jun '16

08 Jun '16

Dear all, Some Hadamard products are involved in my equations, and I did not find a way to implement it. The idea is to apply a element by element product between two matrices A and B, where : - A and B have the same size, let say tensor of dim (N,) - M = A*B is also a tensor of dim (N,) where each elements M[i] = A[i]*B[i] (not summed!) ; (same with rank 2 tensors) So I would like to use the same kind of product as the dot product, but without summing the elements. Is that possible ? Thanks, Floriane

2 2

Re: [firedrake] Wrong matrix assembly
by Fabio Luporini 08 Jun '16

08 Jun '16

Hello Anna, the code is not accessible as it appears to be in a private repository. Could you fix this? Thanks -- Fabio 2016-06-08 14:10 GMT+02:00 Anna Kalogirou <a.kalogirou(a)leeds.ac.uk>: > Dear all, > > I have a simple question regarding a matrix assembly in Firedrake which I > believe is not computed correctly. > > The relevant code can be found here > <https://bitbucket.org/annakalog/buoy2d/src/14334d3c20b9f10ed7c1246cde9e3cb6…> > and the issue is with the B_mu matrix in solvers.py, line 51. I compare the > result with my own Matlab code and the two don't agree. Also, the rest of > the matrices M, A, C_lam agree in both sets of code. > > Can someone help with what the problem might be? > > Thanks, > > Anna. > > -- > > > Dr Anna Kalogirou > Research Fellow > School of Mathematics > University of Leeds > > http://www1.maths.leeds.ac.uk/~matak/ > >

2 1

Wrong matrix assembly
by Anna Kalogirou 08 Jun '16

08 Jun '16

Dear all, I have a simple question regarding a matrix assembly in Firedrake which I believe is not computed correctly. The relevant code can be found here <https://bitbucket.org/annakalog/buoy2d/src/14334d3c20b9f10ed7c1246cde9e3cb6…> and the issue is with the B_mu matrix in solvers.py, line 51. I compare the result with my own Matlab code and the two don't agree. Also, the rest of the matrices M, A, C_lam agree in both sets of code. Can someone help with what the problem might be? Thanks, Anna. -- Dr Anna Kalogirou Research Fellow School of Mathematics University of Leeds http://www1.maths.leeds.ac.uk/~matak/

1 0

Saving txt files while running in parallel
by Floriane Gidel [RPG] 08 Jun '16

08 Jun '16

Dear all, I am running a Firedrake code in parallel with 4 cores, in which I save the maximal value of the amplitude of the wave at each time, with the command: max_eta = max(eta.dat.data) Eta_file.write('%-10s %-10s %-10s\n' % (t,'', eta_max)) But when opening the .txt file, I notice two issues: - the data are not saved continuously: time goes from 0 to 100 (let's say) and then starts from 90 again; - the values of max_eta are not the maximal values of eta, except at the beginning. So it looks like the function max takes the maximum value of eta only in one subdomain, so that after the wave has crossed the subdomain, the maximal value of eta goes back to the depth of rest value. How can I force this command to be applied on the full domain, even if the code is run in parallel? Thanks, Floriane

2 1

Gusto site up
by David Ham 07 Jun '16

07 Jun '16

Hi all, as requested, the Gusto site is up. If Colin provides me with a public key, I'll set things up so he can push updates Cheers, David

1 0

Fwd: [AMCG-Seminars] AMCG seminar on THURSDAY 9th June 4pm G38!
by Lawrence Mitchell 06 Jun '16

06 Jun '16

Dear all, this thursday Mark Adams will be giving an interesting talk on a new benchmarking metric for HPC systems, HPGMG. I'm sure it will be worthwhile attending! Lawrence > Begin forwarded message: > > From: Matthew Piggott <m.d.piggott(a)imperial.ac.uk> > Subject: Fwd: [AMCG-Seminars] AMCG seminar on THURSDAY 9th June 4pm G38! > Date: 6 June 2016 at 13:47:04 BST > To: Paul Kelly <p.kelly(a)imperial.ac.uk>, David Ham <david.ham(a)imperial.ac.uk>, "Mitchell, Lawrence" <lawrence.mitchell(a)imperial.ac.uk> > > You may want to advertise this to the appropriate lists. > Venue is G38 in the RSM building. > Best wishes > Matt > > > ---------- Forwarded message ---------- > From: Derrick, James <james.derrick10(a)imperial.ac.uk <mailto:james.derrick10@imperial.ac.uk>> > Date: 6 June 2016 at 11:46 > Subject: [AMCG-Seminars] AMCG seminar on THURSDAY 9th June 4pm G38! > To: amcg-seminars <amcg-seminars(a)imperial.ac.uk <mailto:amcg-seminars@imperial.ac.uk>> > > > Hi all, > > > This week's seminar will be on Thursday the 9th June (a day early this week) in G38, at the normal time of 4pm where Mark Adams will be giving a talk titled > > > "High Performance Geometric Multigrid: A Supercomputer Benchmark and Metric". > > > The abstract is below. > > Mark Adams and Sam Williams > > The HPGMG project (hpgmg.org <http://www.hpgmg.org/>) develops the extreme-scale benchmark and metric - HPGMG - which is a compact, stand alone, conceptually simple, benchmark, modeled in many ways on HPL. HPGMG solves a system of linear algebraic equations; the metric is equations solved per second (HPL maps equations to floating operations to generate the familiar flops/sec metric). HPGMG solves a finite volume discretization of the Laplacian and uses an appropriate solution algorithm - the non-iterative form of multigrid (full multigrid). HPGMG is designed for use as both a ranking metric and an aid to computer designers for developing architectures, and to centers for configuring systems that are well balanced with respect to general sparse applications. We publish the list twice each year (during ISC and SC). We present a preview of the ISC16 list, which shows some rank reordering relative to the Top500 list, and includes the first GPU (MPI+OpenMP+CUDA) results running on Titan and Piz Daint. > > > See you there! > > James & Roan > > > > > _______________________________________________ > AMCG and Friends Seminar Series Mailing List > AMCG-Seminars(a)imperial.ac.uk > Subscription Options: https://mailman.ic.ac.uk/mailman/listinfo/amcg-seminars > Webpage: http://amcg.ese.ic.ac.uk/index.php?title=Seminar_Series

1 0

PETSc Segmentation Violation
by Anna Kalogirou 06 Jun '16

06 Jun '16

Dear all, I am trying to run a code which returns the following error, before even entering the time loop: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- I tracked down the problem and I found that it is caused by the following: Matrix = assemble(u*v*dx).M.handle where u is a trial and v is a test function. Can any of you help with what might be the issue here? Thank you, Anna. -- Dr Anna Kalogirou Research Fellow School of Mathematics University of Leeds http://www1.maths.leeds.ac.uk/~matak/

5 9

Re: [firedrake] direct solver for mixed space runs out of memory
by Andrew McRae 03 Jun '16

03 Jun '16

Your FunctionSpace W has roughly 388*584*2 = 450k degrees of freedom. Your VectorFunctionSpace V has roughly 2*450k = 900k degrees of freedom. Then you construct a MixedFunctionSpace Q = W*W*V, which will have roughly 1.8 million degrees of freedom. LU causes fill-in, so (unless there's something very special about your PDE/discretisation) each row of the matrix will have O(sqrt(N)) [I think] non-zero entries which must be stored. And 1.8 million * O(sqrt(1.8 million)) * (8 bytes/double + 4 bytes/index) is probably more than 8 GB. Using MUMPS rather than the inbuilt LU might help? I.e., add the option "pc_factor_mat_solver_package": "mumps" to your solver_parameters dict. On 3 June 2016 at 23:15, Tianjiao Sun <tianjiao.sun14(a)imperial.ac.uk> wrote: > Hi team, > > I have the attached python code which is a mixed formulation on extruded > mesh. It is possible that there are other bugs in my formulation, but > I'm running into problems using the non-linear solver. My intent is to > use direct solver in the non-linear iteration, and my parameters are: > > solver_parameters={'ksp_type': 'preonly', 'pc_type': 'lu', > 'snes_atol':1e-10, 'ksp_atol':1e-10}) > > Then PETSc complains that I couldn't use direct solver for nested > matrix. Google guides me to a post by Christian and Lawrence > > https://github.com/firedrakeproject/firedrake/issues/431 > > So I added nest=False in solve(), but then the system run out of memory: > > [0] Out of memory. Allocated: 0, Used by process: 5061259264 > [0] Memory requested 8589934188 > > I guess it's not quite likely to use 8 GB of RAM for this problem size > (388 x 584 x 2 layers). Lawrence did say that "it doesn't work with > vector function spaces" earlier in the post, I'm not sure if that is > related? > > My mixed space is [W, W, V] > where W = DG(0) x CG(1), V is vector space of dim=2. > > Many thanks, > -TJ >

2 1