October 2014 - firedrake - imperial.ac.uk

Firedrake/PyOP2 and GPUs
by Fabio Luporini 18 Oct '14

18 Oct '14

Dear Firedrakers, I'm recruiting a MEng4 student who is potentially interested in adapting COFFEE to GPUs, particularly studying the impact of the available code transformations and exploring new optimisation strategies. Am I right at saying that for running on GPUs some FFC kernels generated through a Firedrake program there is theoretically no problem, provided that: * there's no call to PETSc * no need to handle boundary conditions (for which I guess one needs PyOP2 subsets? Don't know, this is really not my territory...) * what else? In other words, for a simple, trivial program (i.e. I'm not solving anything here): TrialFunctions TestFunctions Coefficients, if any LHS = ... RHS = ... LHS.M RHS._dat_forceevaluation # don't remember the syntax for this, but I guess you can get what I mean offloading the generated kernels to a GPU and collecting back the results should not be a big issue, right? Thanks, and have a great weekend -- Fabio

2 1

Re: [firedrake] Fwd: Re: Firedrake on Maths Cluster
by Andrew McRae 16 Oct '14

16 Oct '14

Correct. Notice that Travis <https://travis-ci.org/firedrakeproject/firedrake/builds> is throwing the same error, too. <a_mcrae> gah, travis is still erroring (during the petsc/python setup?) <wence> yeah, travis is currently borked due to petsc/petsc4py incompatibility <a_mcrae> fuuuuuu <wence> I have local fixes, but no merge request in yet On 16 October 2014 08:21, Cotter, Colin J <colin.cotter(a)imperial.ac.uk> wrote: > Dear all, > Andy Thomas is trying to build Firedrake on the Maths cluster for me, > and seems to be getting some trouble with the petsc installation. Please > could you take a look and see what might be wrong? > Cheers > Cjc > ---------- Forwarded message ---------- > From: "andy thomas" <andy.thomas(a)imperial.ac.uk> > Date: 16 Oct 2014 07:31 > Subject: Re: Firedrake on Maths Cluster > To: "Colin Cotter" <colin.cotter(a)imperial.ac.uk> > Cc: > > On Tue, 14 Oct 2014, Colin Cotter wrote: > > On 14/10/14 10:40, andy thomas wrote: >> >>> >>> Knee-jerk reaction to this suggestion is "B*LLS!" because I spent hours >>> trying to do this back in July. But I'll put it on my ever-lengthening list >>> of things to do - it's on a UROP server so should now be easy to replicate >>> from that. >>> >>> cheers, Andy >>> >> Hi Andy, >> I'll put you in touch with some Firedrake people who can help - it's >> best if you talk to them rather than trying to do it on your own. We also >> will shortly be getting Tim Greaves who is an AMCG system support person >> who's job will shortly become making sure Firedrake runs on things so >> you'll be able to talk to him. >> > > As predicted, n*2 hours have been spent trying to install PETSc on two > sample systems and it keeps bombing out with the error: > > ImportError: No module named install > > There is indeed no module called install in the Python 2.7 installation so > I'm guessing it's created in /tmp as part of the build process but is > destroyed as part of the clean-up after the install fails. > > Any ideas or could you please pass this on to someone who may know the > answer? > > cheers, Andy > > Some relevant info: > > 1) PYTHONPATH: /usr/local/lib/python2.7/dist-packages:/usr/lib/python2.7 > > 2) PETSc installation invoked with: > > PETSC_CONFIGURE_OPTIONS="--download-ctetgen --download-triangle > --download-chaco --with-threadcomm --with-openmp --with-pthreadclasses" pip > install -U --no-deps https://bitbucket.org/petsc/petsc/get/master.tar.bz2 > > 3) tail end of error report: > > Copying petsc.egg-info to /usr/local/lib/python2.7/dist- > packages/petsc-3.6.dev0.egg-info > > running install_scripts > > writing list of installed files to '/tmp/pip-HbDw3I-record/ > install-record.txt' > > PETSc: install > > install options: > > --destDir=/usr/local/lib/python2.7/dist-packages/petsc > > --prefix=/usr/local/lib/python2.7/dist-packages/petsc > > Traceback (most recent call last): > > File "<string>", line 1, in <module> > > File "/tmp/pip-lrwh8t-build/setup.py", line 307, in <module> > > **metadata) > > File "/usr/lib/python2.7/distutils/core.py", line 151, in setup > > dist.run_commands() > > File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands > > self.run_command(cmd) > > File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command > > cmd_obj.run() > > File "/tmp/pip-lrwh8t-build/setup.py", line 221, in run > > install(dest_dir, prefix, self.dry_run) > > File "/tmp/pip-lrwh8t-build/setup.py", line 160, in install > > import install > > ImportError: No module named install > > ---------------------------------------- > Rolling back uninstall of petsc > Cleaning up... > Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip- > lrwh8t-build/setup.py';exec(compile(getattr(tokenize, 'open', > open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install > --record /tmp/pip-HbDw3I-record/install-record.txt > --single-version-externally-managed --compile failed with error code 1 in > /tmp/pip-lrwh8t-build > Traceback (most recent call last): > File "/usr/bin/pip", line 9, in <module> > load_entry_point('pip==1.5.4', 'console_scripts', 'pip')() > File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 185, in > main > return command.main(cmd_args) > File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 161, > in main > text = '\n'.join(complete_log) > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 68: > ordinal not in range(128) > >

1 0

Fwd: Re: Firedrake on Maths Cluster
by Colin Cotter 16 Oct '14

16 Oct '14

Dear all, Andy Thomas is trying to build Firedrake on the Maths cluster for me, and seems to be getting some trouble with the petsc installation. Please could you take a look and see what might be wrong? Cheers Cjc ---------- Forwarded message ---------- From: "andy thomas" <andy.thomas(a)imperial.ac.uk> Date: 16 Oct 2014 07:31 Subject: Re: Firedrake on Maths Cluster To: "Colin Cotter" <colin.cotter(a)imperial.ac.uk> Cc: On Tue, 14 Oct 2014, Colin Cotter wrote: On 14/10/14 10:40, andy thomas wrote: > >> >> Knee-jerk reaction to this suggestion is "B*LLS!" because I spent hours >> trying to do this back in July. But I'll put it on my ever-lengthening list >> of things to do - it's on a UROP server so should now be easy to replicate >> from that. >> >> cheers, Andy >> > Hi Andy, > I'll put you in touch with some Firedrake people who can help - it's > best if you talk to them rather than trying to do it on your own. We also > will shortly be getting Tim Greaves who is an AMCG system support person > who's job will shortly become making sure Firedrake runs on things so > you'll be able to talk to him. > As predicted, n*2 hours have been spent trying to install PETSc on two sample systems and it keeps bombing out with the error: ImportError: No module named install There is indeed no module called install in the Python 2.7 installation so I'm guessing it's created in /tmp as part of the build process but is destroyed as part of the clean-up after the install fails. Any ideas or could you please pass this on to someone who may know the answer? cheers, Andy Some relevant info: 1) PYTHONPATH: /usr/local/lib/python2.7/dist-packages:/usr/lib/python2.7 2) PETSc installation invoked with: PETSC_CONFIGURE_OPTIONS="--download-ctetgen --download-triangle --download-chaco --with-threadcomm --with-openmp --with-pthreadclasses" pip install -U --no-deps https://bitbucket.org/petsc/petsc/get/master.tar.bz2 3) tail end of error report: Copying petsc.egg-info to /usr/local/lib/python2.7/dist- packages/petsc-3.6.dev0.egg-info running install_scripts writing list of installed files to '/tmp/pip-HbDw3I-record/ install-record.txt' PETSc: install install options: --destDir=/usr/local/lib/python2.7/dist-packages/petsc --prefix=/usr/local/lib/python2.7/dist-packages/petsc Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-lrwh8t-build/setup.py", line 307, in <module> **metadata) File "/usr/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/tmp/pip-lrwh8t-build/setup.py", line 221, in run install(dest_dir, prefix, self.dry_run) File "/tmp/pip-lrwh8t-build/setup.py", line 160, in install import install ImportError: No module named install ---------------------------------------- Rolling back uninstall of petsc Cleaning up... Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip- lrwh8t-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-HbDw3I-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip-lrwh8t-build Traceback (most recent call last): File "/usr/bin/pip", line 9, in <module> load_entry_point('pip==1.5.4', 'console_scripts', 'pip')() File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 185, in main return command.main(cmd_args) File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 161, in main text = '\n'.join(complete_log) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 68: ordinal not in range(128)

1 0

Re: [firedrake] Banner draft
by David Ham 14 Oct '14

14 Oct '14

Hi All, Thanks for the input, I'll caption the figures and replace the EPSRC logo. The styling is more tricky since it's set by Chris Cantwell's template which all the PRISM participants are following, but I can get away with a minor font change. David On Tuesday, October 14, 2014, Rathgeber, Florian < florian.rathgeber(a)imperial.ac.uk> wrote: > On 14/10/14 20:02, Lawrence Mitchell wrote: > > > > On 14 Oct 2014, at 19:18, David Ham <David.Ham(a)imperial.ac.uk > <javascript:;>> wrote: > > > >> Greetings firedrakers, > >> > >> There is a stakeholder meeting for the PRISM grant (big Imperial finite > element grant) next week for which I need to provide a roll-up banner > giving a very high level overview of the project. I should really, really > fire off the banner to be printed in the morning, so any rapid feedback > would be appreciated: > >> > >> http://www.doc.ic.ac.uk/~dham/Prism_Banner_Firedrake.pdf > > > > > > Content looks good, excepting the bits of pixelation that probably > no-one but me will ever notice/care about. > > Agree to both. The font is not the prettiest (I assume sans?), maybe try > Open Sans, Droid Sans or Roboto instead? The URLs at the bottom look, > not sure what to do about them, maybe try a drop shadow? > > Florian > > > FWIW EPSRC do provide vector versions of their logos, e.g.: > > > > EPSRC logo as EPS: > http://www.epsrc.ac.uk/files/aboutus/logos-and-indentity/colour-master-logo… > > > > Lawrence > > -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

1 0

Banner draft
by David Ham 14 Oct '14

14 Oct '14

Greetings firedrakers, There is a stakeholder meeting for the PRISM grant (big Imperial finite element grant) next week for which I need to provide a roll-up banner giving a very high level overview of the project. I should really, really fire off the banner to be printed in the morning, so any rapid feedback would be appreciated: http://www.doc.ic.ac.uk/~dham/Prism_Banner_Firedrake.pdf David -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

5 5

Re: [firedrake] Banner draft
by Fabio Luporini 14 Oct '14

14 Oct '14

Great. Can you indent the pragma to the k loop? 2014-10-14 20:15 GMT+01:00 Mitchell, Lawrence < lawrence.mitchell(a)imperial.ac.uk>: > > On 14 Oct 2014, at 20:05, Marie E. Rognes <meg(a)simula.no> wrote: > > > > > This looks extremely nice! What are the simulations (bottom two plots)? > > The sphere one is the nonlinear shallow water flow past a mountain test > case (aka Williamson 5) [Williamson, D. L., Drake, J. B., Hack, J. J., > Jakob, R., and Swarz- trauber, P. N.: A standard test set for numerical > approximations to the shallow water equations in spherical geometry, J. > Computat. Phys., 102, 211–224, 1992] > > I guess the bottom one is N-S flow past a rectangle? > > Lawrence >

1 0

Re: [firedrake] Firedrake paper results
by David Ham 12 Oct '14

12 Oct '14

In figure 2, the RHS overhead looks awfully high. It's an order of magnitude worse than that in figure 5. Is this correct? David On 11 October 2014 11:27, Rathgeber, Florian < florian.rathgeber(a)imperial.ac.uk> wrote: > I'd still be interested in feedback on this! > > On 04/10/14 09:12, Florian Rathgeber wrote: > > Dear all, > > > > I finished a first draft of the results section for the Firedrake paper. > > Any feedback gratefully received! > > > > If you don't have access to the repository you can get a PDF from > > > > https://wwwhomes.doc.ic.ac.uk/~fr710/paper.pdf > > > > Cheers, > > Florian > > -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

2 1

Re: [firedrake] Chemora: A PDE Solving Framework for Modern HPC Architectures
by Fabio Luporini 12 Oct '14

12 Oct '14

I've just skimmed through the paper, it seems they share lots of high-level concepts with Firadrake/FEniCS, although it's weird they don't systematically compare to these approaches (OK, fine for Firedrake that is a new system, but FEniCS...) As for the code optimizations: the loop fission thing is interesting, although it's exactly the work the Carlo and OP2 people did in OP2. And they don't say the most important aspects: *when* they have to apply that and what "semi-automatically" means. As for the "explicit" vectorization, I'm really surprised a compiler can't do what they claim, perhaps that's why they don't even mention how they compiled the code, as far as I saw (btw, they also contradict themselves, since they also say "Since Kranc generates out- put in a high-level language (e.g. C++), it does not have to deal with very low-level architecture details such as register allocation or instruction selection." It would be interesting to know more. -- Fabio 2014-10-11 20:01 GMT+01:00 McRae, Andrew <a.mcrae12(a)imperial.ac.uk>: > "...The two most important optimisations were loop fission to not > overflow the instruction cache, and manual vectorisation..." > > Sounds like they have their own Fabio! > > On 11 October 2014 11:22, Rathgeber, Florian < > florian.rathgeber(a)imperial.ac.uk> wrote: > >> In Chemora, PDEs are expressed either in a high-level LaTeX-like >> language or in Mathematica. It's based on Cactus: >> >> http://arxiv.org/abs/1410.1764 >> >> Florian >> >> >

1 0

Re: [firedrake] Chemora: A PDE Solving Framework for Modern HPC Architectures
by Andrew McRae 12 Oct '14

12 Oct '14

"...The two most important optimisations were loop fission to not overflow the instruction cache, and manual vectorisation..." Sounds like they have their own Fabio! On 11 October 2014 11:22, Rathgeber, Florian < florian.rathgeber(a)imperial.ac.uk> wrote: > In Chemora, PDEs are expressed either in a high-level LaTeX-like > language or in Mathematica. It's based on Cactus: > > http://arxiv.org/abs/1410.1764 > > Florian > >

2 1

firedrake on ARCHER: PETSc issue
by Eike Mueller 11 Oct '14

11 Oct '14

Dear firedraker, after installing PETSc and petsc4py in my own $WORK, I can run sequentially, but if I run on more than one core it hangs when importing firdrake. I traced this down to the call of PETSc._initialise(args, comm) in the method init() in petsc4py/build/lib.linux-x86_64-2.7/petsc4py/__init__.py, which just does not return. It does pick up my PETSc installation correctly (printed out path, arch in ImportPETSc method in lib/__init__.py). I build the PETSc branch mlange/plex-distributed-overlap (same as in $FDRAKE_DIR) with the same configure options use there, and then I build petsc4py with make. Any ideas? Thanks, Eike -- Dr Eike Hermann Mueller Research Associate (PostDoc) Department of Mathematical Sciences University of Bath Bath BA2 7AY, United Kingdom +44 1225 38 5803 e.mueller(a)bath.ac.uk http://people.bath.ac.uk/em459/

3 4