- firedrake - imperial.ac.uk

PETSc install instructions update
by David Ham 19 Oct '14

19 Oct '14

Dear Firedrakers, Some of you have recently had problems with installing PETSc recently. There is now a fix for this, and also new install URLs for PETSc and petsc4py which will hopefully reduce the number of ongoing PETSc install issues that people have. The short version is that the instructions for installing PETSc and petsc4py on our websites have been updated, so if you are installing or upgrading PETSc you should follow the new instructions: http://www.firedrakeproject.org/obtaining_pyop2.html#petsc The ever so slightly longer version is that we now point to our own fork of PETSc rather than the upstream master. We will update our forks as necessary and only after checking that the version we update to appears to work. This should hopefully reduce the number of random breakages users encounter in PETSc. Cheers, David -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

3 3

Re: [firedrake] firedrake on ARCHER - helmholtz solver for larger core counts
by Andrew McRae 18 Oct '14

18 Oct '14

Question to others: could this be related to the 'subtract a large number to apply BCs' hack, or is that only for extruded meshes? On 18 Oct 2014 11:43, "Eike Mueller" <e.mueller(a)bath.ac.uk> wrote: > Dear all, > > with Lawrence's fix it now works with 10 levels (20,971,520 cells) on > two full nodes. If I increase both the number of cells and the number of > nodes by a factor of 4 (i.e. same #dof/core) I get a crash again. However, > the position in the code where it crashes varies, here are two example: > > https://gist.github.com/eikehmueller/a07007af72b34a57efff > > In the 1st and 3rd run, I had cleared the PyOP2/firedrake cashes, > whereas in the second case I'm not entirely sure what the state of the > cache was. > In the 3rd run I also set PYOP2_DEBUG=1, whereas it was 0 in the first two > runs. > The weird thing is also that the 3rd run seems to complete, but still > generates the dreaded PETSc seg fault. > > Thanks, > > Eike > > > -- > > Dr Eike Hermann Mueller > Research Associate (PostDoc) > > Department of Mathematical Sciences > University of Bath > Bath BA2 7AY, United Kingdom > > +44 1225 38 5803 > e.mueller(a)bath.ac.uk > http://people.bath.ac.uk/em459/ > > On 13 Oct 2014, at 11:22, Eike Mueller <E.Mueller(a)bath.ac.uk> wrote: > > Dear all, > > I rebuilt 'my' PETSc with debugging enabled, and then also rebuilt > petsc4py and firedrake. I could run on 48 cores with 7 levels once, but > this was not reproducible. When I tried again it crashed with a PETSc error. > > Is there any chance that the branches below help? Do I need > mlange/plex-distributed-overlap or can I use the PETSc master now? > > Thanks a lot, > > Eike > > firedrake: multigrid-parallel > pyop2: local-par_loop > petsc4py: bitbucket.org/mapdes/petsc4py branch moar-plex > petsc: mlange/plex-distributed-overlap > > > Functionality similar to the latter should hopefully arrive in petsc > master this week > > Lawrence > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > > -- > Dr Eike Hermann Mueller > Research Associate (PostDoc) > > Department of Mathematical Sciences > University of Bath > Bath BA2 7AY, United Kingdom > > +44 1225 38 5803 > e.mueller(a)bath.ac.uk > http://people.bath.ac.uk/em459/ > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > >

1 0

firedrake on ARCHER - helmholtz solver for larger core counts
by Eike Mueller 18 Oct '14

18 Oct '14

Dear firedrakers, I tried to run the Helmholtz multigrid solver on higher core counts on ARCHER. In certain configurations it works, but in others it crashes. The number of fine grid cells is 20*4^(4+refcount_coarse). In particular, I was not able to run on 48 cores (i.e. 2 full nodes) 1 node, 16 cores/node = 16 cores, refcount_coarse=5: Runs 2 nores, 16 cores/node = 48 cores, refcount_coarse=4: hangs without printing error message 2 nores, 16 cores/node = 48 cores, refcount_coarse=5: crashes with PETSc error message 2 nores, 16 cores/node = 48 cores, refcount_coarse=5: crashes with different error message Output logs for all cases are attached, as well as the pbs scripts for running on 48 cores. I also tried running the poisson example in firedrake-benchmark in different configurations: 1 node, 16 cores/node = 16 cores => works 2 nodes, 8 cores/node = 16 cores => works 2 nodes, 24 cores/node = 48 cores => crashes with PETSc error I did use the generic ARCHER firedrake installation in this case, though. In the Helmholtz solver runs I used my own version of firedrake and PyOP2 since I need the multigrid and local_par-loop branches. This makes me suspicious that it is not a problem specific to my code, but then I thought that you did run firedrake on relatively large core counts before, didn't you? The output of the Poisson benchmarks is also attached. Any ideas what might be going wrong and what is the best way for debugging this? Thanks, Eike -- Dr Eike Hermann Mueller Research Associate (PostDoc) Department of Mathematical Sciences University of Bath Bath BA2 7AY, United Kingdom +44 1225 38 5803 e.mueller(a)bath.ac.uk http://people.bath.ac.uk/em459/

3 8

Firedrake/PyOP2 and GPUs
by Fabio Luporini 18 Oct '14

18 Oct '14

Dear Firedrakers, I'm recruiting a MEng4 student who is potentially interested in adapting COFFEE to GPUs, particularly studying the impact of the available code transformations and exploring new optimisation strategies. Am I right at saying that for running on GPUs some FFC kernels generated through a Firedrake program there is theoretically no problem, provided that: * there's no call to PETSc * no need to handle boundary conditions (for which I guess one needs PyOP2 subsets? Don't know, this is really not my territory...) * what else? In other words, for a simple, trivial program (i.e. I'm not solving anything here): TrialFunctions TestFunctions Coefficients, if any LHS = ... RHS = ... LHS.M RHS._dat_forceevaluation # don't remember the syntax for this, but I guess you can get what I mean offloading the generated kernels to a GPU and collecting back the results should not be a big issue, right? Thanks, and have a great weekend -- Fabio

2 1

Re: [firedrake] Fwd: Re: Firedrake on Maths Cluster
by Andrew McRae 16 Oct '14

16 Oct '14

Correct. Notice that Travis <https://travis-ci.org/firedrakeproject/firedrake/builds> is throwing the same error, too. <a_mcrae> gah, travis is still erroring (during the petsc/python setup?) <wence> yeah, travis is currently borked due to petsc/petsc4py incompatibility <a_mcrae> fuuuuuu <wence> I have local fixes, but no merge request in yet On 16 October 2014 08:21, Cotter, Colin J <colin.cotter(a)imperial.ac.uk> wrote: > Dear all, > Andy Thomas is trying to build Firedrake on the Maths cluster for me, > and seems to be getting some trouble with the petsc installation. Please > could you take a look and see what might be wrong? > Cheers > Cjc > ---------- Forwarded message ---------- > From: "andy thomas" <andy.thomas(a)imperial.ac.uk> > Date: 16 Oct 2014 07:31 > Subject: Re: Firedrake on Maths Cluster > To: "Colin Cotter" <colin.cotter(a)imperial.ac.uk> > Cc: > > On Tue, 14 Oct 2014, Colin Cotter wrote: > > On 14/10/14 10:40, andy thomas wrote: >> >>> >>> Knee-jerk reaction to this suggestion is "B*LLS!" because I spent hours >>> trying to do this back in July. But I'll put it on my ever-lengthening list >>> of things to do - it's on a UROP server so should now be easy to replicate >>> from that. >>> >>> cheers, Andy >>> >> Hi Andy, >> I'll put you in touch with some Firedrake people who can help - it's >> best if you talk to them rather than trying to do it on your own. We also >> will shortly be getting Tim Greaves who is an AMCG system support person >> who's job will shortly become making sure Firedrake runs on things so >> you'll be able to talk to him. >> > > As predicted, n*2 hours have been spent trying to install PETSc on two > sample systems and it keeps bombing out with the error: > > ImportError: No module named install > > There is indeed no module called install in the Python 2.7 installation so > I'm guessing it's created in /tmp as part of the build process but is > destroyed as part of the clean-up after the install fails. > > Any ideas or could you please pass this on to someone who may know the > answer? > > cheers, Andy > > Some relevant info: > > 1) PYTHONPATH: /usr/local/lib/python2.7/dist-packages:/usr/lib/python2.7 > > 2) PETSc installation invoked with: > > PETSC_CONFIGURE_OPTIONS="--download-ctetgen --download-triangle > --download-chaco --with-threadcomm --with-openmp --with-pthreadclasses" pip > install -U --no-deps https://bitbucket.org/petsc/petsc/get/master.tar.bz2 > > 3) tail end of error report: > > Copying petsc.egg-info to /usr/local/lib/python2.7/dist- > packages/petsc-3.6.dev0.egg-info > > running install_scripts > > writing list of installed files to '/tmp/pip-HbDw3I-record/ > install-record.txt' > > PETSc: install > > install options: > > --destDir=/usr/local/lib/python2.7/dist-packages/petsc > > --prefix=/usr/local/lib/python2.7/dist-packages/petsc > > Traceback (most recent call last): > > File "<string>", line 1, in <module> > > File "/tmp/pip-lrwh8t-build/setup.py", line 307, in <module> > > **metadata) > > File "/usr/lib/python2.7/distutils/core.py", line 151, in setup > > dist.run_commands() > > File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands > > self.run_command(cmd) > > File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command > > cmd_obj.run() > > File "/tmp/pip-lrwh8t-build/setup.py", line 221, in run > > install(dest_dir, prefix, self.dry_run) > > File "/tmp/pip-lrwh8t-build/setup.py", line 160, in install > > import install > > ImportError: No module named install > > ---------------------------------------- > Rolling back uninstall of petsc > Cleaning up... > Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip- > lrwh8t-build/setup.py';exec(compile(getattr(tokenize, 'open', > open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install > --record /tmp/pip-HbDw3I-record/install-record.txt > --single-version-externally-managed --compile failed with error code 1 in > /tmp/pip-lrwh8t-build > Traceback (most recent call last): > File "/usr/bin/pip", line 9, in <module> > load_entry_point('pip==1.5.4', 'console_scripts', 'pip')() > File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 185, in > main > return command.main(cmd_args) > File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 161, > in main > text = '\n'.join(complete_log) > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 68: > ordinal not in range(128) > >

1 0

Fwd: Re: Firedrake on Maths Cluster
by Colin Cotter 16 Oct '14

16 Oct '14

Dear all, Andy Thomas is trying to build Firedrake on the Maths cluster for me, and seems to be getting some trouble with the petsc installation. Please could you take a look and see what might be wrong? Cheers Cjc ---------- Forwarded message ---------- From: "andy thomas" <andy.thomas(a)imperial.ac.uk> Date: 16 Oct 2014 07:31 Subject: Re: Firedrake on Maths Cluster To: "Colin Cotter" <colin.cotter(a)imperial.ac.uk> Cc: On Tue, 14 Oct 2014, Colin Cotter wrote: On 14/10/14 10:40, andy thomas wrote: > >> >> Knee-jerk reaction to this suggestion is "B*LLS!" because I spent hours >> trying to do this back in July. But I'll put it on my ever-lengthening list >> of things to do - it's on a UROP server so should now be easy to replicate >> from that. >> >> cheers, Andy >> > Hi Andy, > I'll put you in touch with some Firedrake people who can help - it's > best if you talk to them rather than trying to do it on your own. We also > will shortly be getting Tim Greaves who is an AMCG system support person > who's job will shortly become making sure Firedrake runs on things so > you'll be able to talk to him. > As predicted, n*2 hours have been spent trying to install PETSc on two sample systems and it keeps bombing out with the error: ImportError: No module named install There is indeed no module called install in the Python 2.7 installation so I'm guessing it's created in /tmp as part of the build process but is destroyed as part of the clean-up after the install fails. Any ideas or could you please pass this on to someone who may know the answer? cheers, Andy Some relevant info: 1) PYTHONPATH: /usr/local/lib/python2.7/dist-packages:/usr/lib/python2.7 2) PETSc installation invoked with: PETSC_CONFIGURE_OPTIONS="--download-ctetgen --download-triangle --download-chaco --with-threadcomm --with-openmp --with-pthreadclasses" pip install -U --no-deps https://bitbucket.org/petsc/petsc/get/master.tar.bz2 3) tail end of error report: Copying petsc.egg-info to /usr/local/lib/python2.7/dist- packages/petsc-3.6.dev0.egg-info running install_scripts writing list of installed files to '/tmp/pip-HbDw3I-record/ install-record.txt' PETSc: install install options: --destDir=/usr/local/lib/python2.7/dist-packages/petsc --prefix=/usr/local/lib/python2.7/dist-packages/petsc Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-lrwh8t-build/setup.py", line 307, in <module> **metadata) File "/usr/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/tmp/pip-lrwh8t-build/setup.py", line 221, in run install(dest_dir, prefix, self.dry_run) File "/tmp/pip-lrwh8t-build/setup.py", line 160, in install import install ImportError: No module named install ---------------------------------------- Rolling back uninstall of petsc Cleaning up... Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip- lrwh8t-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-HbDw3I-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip-lrwh8t-build Traceback (most recent call last): File "/usr/bin/pip", line 9, in <module> load_entry_point('pip==1.5.4', 'console_scripts', 'pip')() File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 185, in main return command.main(cmd_args) File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 161, in main text = '\n'.join(complete_log) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 68: ordinal not in range(128)

1 0

Re: [firedrake] Banner draft
by David Ham 14 Oct '14

14 Oct '14

Hi All, Thanks for the input, I'll caption the figures and replace the EPSRC logo. The styling is more tricky since it's set by Chris Cantwell's template which all the PRISM participants are following, but I can get away with a minor font change. David On Tuesday, October 14, 2014, Rathgeber, Florian < florian.rathgeber(a)imperial.ac.uk> wrote: > On 14/10/14 20:02, Lawrence Mitchell wrote: > > > > On 14 Oct 2014, at 19:18, David Ham <David.Ham(a)imperial.ac.uk > <javascript:;>> wrote: > > > >> Greetings firedrakers, > >> > >> There is a stakeholder meeting for the PRISM grant (big Imperial finite > element grant) next week for which I need to provide a roll-up banner > giving a very high level overview of the project. I should really, really > fire off the banner to be printed in the morning, so any rapid feedback > would be appreciated: > >> > >> http://www.doc.ic.ac.uk/~dham/Prism_Banner_Firedrake.pdf > > > > > > Content looks good, excepting the bits of pixelation that probably > no-one but me will ever notice/care about. > > Agree to both. The font is not the prettiest (I assume sans?), maybe try > Open Sans, Droid Sans or Roboto instead? The URLs at the bottom look, > not sure what to do about them, maybe try a drop shadow? > > Florian > > > FWIW EPSRC do provide vector versions of their logos, e.g.: > > > > EPSRC logo as EPS: > http://www.epsrc.ac.uk/files/aboutus/logos-and-indentity/colour-master-logo… > > > > Lawrence > > -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

1 0

Banner draft
by David Ham 14 Oct '14

14 Oct '14

Greetings firedrakers, There is a stakeholder meeting for the PRISM grant (big Imperial finite element grant) next week for which I need to provide a roll-up banner giving a very high level overview of the project. I should really, really fire off the banner to be printed in the morning, so any rapid feedback would be appreciated: http://www.doc.ic.ac.uk/~dham/Prism_Banner_Firedrake.pdf David -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

5 5

Re: [firedrake] Banner draft
by Fabio Luporini 14 Oct '14

14 Oct '14

Great. Can you indent the pragma to the k loop? 2014-10-14 20:15 GMT+01:00 Mitchell, Lawrence < lawrence.mitchell(a)imperial.ac.uk>: > > On 14 Oct 2014, at 20:05, Marie E. Rognes <meg(a)simula.no> wrote: > > > > > This looks extremely nice! What are the simulations (bottom two plots)? > > The sphere one is the nonlinear shallow water flow past a mountain test > case (aka Williamson 5) [Williamson, D. L., Drake, J. B., Hack, J. J., > Jakob, R., and Swarz- trauber, P. N.: A standard test set for numerical > approximations to the shallow water equations in spherical geometry, J. > Computat. Phys., 102, 211–224, 1992] > > I guess the bottom one is N-S flow past a rectangle? > > Lawrence >

1 0

Re: [firedrake] Firedrake paper results
by David Ham 12 Oct '14

12 Oct '14

In figure 2, the RHS overhead looks awfully high. It's an order of magnitude worse than that in figure 5. Is this correct? David On 11 October 2014 11:27, Rathgeber, Florian < florian.rathgeber(a)imperial.ac.uk> wrote: > I'd still be interested in feedback on this! > > On 04/10/14 09:12, Florian Rathgeber wrote: > > Dear all, > > > > I finished a first draft of the results section for the Firedrake paper. > > Any feedback gratefully received! > > > > If you don't have access to the repository you can get a PDF from > > > > https://wwwhomes.doc.ic.ac.uk/~fr710/paper.pdf > > > > Cheers, > > Florian > > -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

2 1