September 2014 - firedrake

Progress on installing PyOP2/firedrake on ARCHER
by Eike Mueller 29 Sep '14

29 Sep '14

Dear firedrakers, I'm just going through the instructions for installing firedrake on ARCHER as per https://github.com/firedrakeproject/firedrake-bench/wiki/Archer . I ran into a couple of issues you might be interested in, so I thought I keep you updated on my progress. The content of my firedrake.env script is listed below and also attached (I use export instead of setenv since I work with the bash shell - I think this is the default on ARCHER, isn't it?). Note that I have to unload the python module, otherwise I get a conflict with anaconda. ======== 'module' command broken by anaconda ======== However, after sourcing firedrake.env the 'module' command does not work any longer: eike@eslogin003 $ module list init.c(479):ERROR:161: Cannot initialize TCL I think the reason is the following addition to LD_LIBRARY_PATH in ${FDRAKE_DIR}/modules/fdrake-python-env prepend-path LD_LIBRARY_PATH "$env(ANACONDA_LIB)" If I remove ${ANACONDA_LIB} from LD_LIBRARY_PATH afterwards (LD_LIBRARY_PATH=`echo $LD_LIBRARY_PATH | sed 's/\/work\/y07\/y07\/cse\/anaconda\/1.9.2\/lib//g'`) the module command works again (I also tried to attend ${ANACONDA_LIB} at the end of LD_LIBRARY_PATH, but then I get the same problem again). ======== Cloning PyOP2 and firedrake repositories from github ======== Also, I could not clone PyOP2 and firedrake via git clone git@github.com:OP2/PyOP2.git git clone git@github.com:firedrakeproject/firedrake.git I get the following error message: eike@eslogin008 $ git clone git@github.com:OP2/PyOP2.git Cloning into 'PyOP2'... Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. That might be an issue with the git configuration on ARCHER, though. I fixed this by just cloning via https: git clone https://github.com/OP2/PyOP2.git git clone https://github.com/firedrakeproject/firedrake.git ======== Building PyOP2 ======== When I try to build PyOP2 with eike@eslogin002 $ python setup.py build_ext --inplace I get the following error message (see attached pyop2_call1.log): [...] File "/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/Cython/Distutils/build_ext.py", line 238, in cython_sources for i in extension.include_dirs: File "setup.py", line 83, in include_dirs from numpy import get_include ImportError: cannot import name get_include This is odd, since 'from numpy import get_include' does work if I try it in the interactive python shell. If I then run 'python setup.py build_ext --inplace' again, the build process seems to go through (pyop2_call2.log). ======== Building firedrake ======== this seems to go though ok (see attached file firedrake.log). ======== Running poisson example code ======== I tried to run the example; my version of the modified firedrake.tpl is attached (notice that I unload the python module to avoid conflicts with anaconda). Unfortunately it doesn't seem to work, I get an error in fd_poisson.0101.541091.sdb.log trying to load the python libraries: python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory All three output files are attached below as well. If this helps, I'm very happy to edit the wiki. Cheers, Eike *** content of firedrake.env *** module swap PrgEnv-cray PrgEnv-gnu module unload python module add anaconda # Add fdrake module path export FDRAKE_DIR=/work/y07/y07/fdrake module use "$FDRAKE_DIR/modules" # Load build environment module load fdrake-build-env module load fdrake-python-env LD_LIBRARY_PATH=`echo $LD_LIBRARY_PATH | sed 's/\/work\/y07\/y07\/cse\/anaconda\/1.9.2\/lib//g'` export PYTHONPATH=$WORK/firedrake-bench:$WORK/firedrake-bench/pybench:$WORK/firedrake:$WORK/PyOP2:$PYTHONPATH -- Dr Eike Hermann Mueller Research Associate (PostDoc) Department of Mathematical Sciences University of Bath Bath BA2 7AY, United Kingdom +44 1225 38 5803 e.mueller(a)bath.ac.uk http://people.bath.ac.uk/em459/

3 10

hedgehogize
by Cotter, Colin J 26 Sep '14

26 Sep '14

Dear all, How do people feel about me adding a hedgehogize utility to Firedrake, that takes an extruded sphere mesh and applies the hedgehog transformation? If agreed, where shall I put it? cheers --cjc

2 2

Re: [firedrake] Firedrake on supercomputers
by Tuomas Karna 24 Sep '14

24 Sep '14

Hi Lawrence, Thanks for these, this is great. On stampede I can access the compilers in the devel queue so this works like a charm. Would you have any hints for improving parallel scalabilty? Right now things start to level out at 8 cores (2d shallow water with p2-p1 elems, 76k triangles). Also, is there another tool for timings than PYOP2_PRINT_SUMMARY? Cheers, Tuomas > Hi Tuomas, > > comments in line below. > > On 7 Aug 2014, at 07:31, Tuomas Karna <tuomas.karna at gmail.com <https://mailman.ic.ac.uk/mailman/listinfo/firedrake>> wrote: > > >/ Hi all, > />/ > />/ I'm running my code on TACC Stampede and I have a couple of questions: > />/ > />/ How can I setup Firedrake/PyOP2 for the target machine, for example to use Intel compilers, target architecture (sandybridge) instruction set, or Intel MKL libraries? > / > Note that for all this to work, you need to be able to launch the compiler on compute nodes (in particular for intel, the compute nodes will have to be able to talk to the license server), but I hope this all works. > > Selection of the intel compiler is: > > from firedrake import * > parameters["coffee"]["compiler"] = "intel" > > This still launches 'mpicc' but uses intel specific flags when compiling. If the compiler is actually called something different (e.g. on Cray systems it's always cc) then you need to do: > > export CC=name_of_cc_compiler > # only do this if the linker is separate from the compiler > export LDSHARED=name_of_linker > > and then: > > from firedrake import * > parameters["coffee"]["compiler"] = "intel" > > In addition, you probably want to select the AVX instruction set: > > parameters["coffee"]["simd_isa"] = "avx" > > Finally, you can select various code transformations to be applied to the generated kernels to increase their performance (seehttp://arxiv.org/abs/1407.0904 for many details). In particular you may wish to try: > > # loop invariant code motion > parameters["coffee"]["licm"] = True > > # Data alignment (to give compilers a better chance at vectorising) > parameters["coffee"]["ap"] = True > > # Request the compiler try harder to auto-vectorise > from pyop2.coffee.ast_plan import AUTOVECT > parameters["coffee"]["vect"] = (AUTOVECT, -1) > > > Our experience is that for most forms, the big gains come from loop invariant code motion and data alignment, but Fabio can comment in much more detail on this. > > Finally, usage of MKL, by default, none of the code Firedrake/PyOP2 generate uses BLAS at all, so if you want to exploit BLAS in the solver library, you just needed to compile PETSc appropriately. You can apply code transformations to the kernels to convert them into BLAS calls, but for low order, it may not be much of a win (since we do not amortise the call-overhead across elements), but you can try it: > > # BLAS-transformations aren't exposed through the parameters dict so: > from firedrake import * > op2.configuration["blas"] = "mkl" > > ... > > You can also, if it's available, try Eigen instead: > > op2.configuration["blas"] = "eigen" > > > >/ Until now I've only used MPI. What do I need to do to run with openmp or cuda for instance? I've got the PyOP2 dependencies in place, as they are listed on the Firedrake website. > / > Running with openmp requires that you've built PETSc "-with-threadcomm --with-openmp --with-pthreadclasses". You can then either do: > > export PYOP2_BACKEND=openmp > # Or however your batch system wants you to do this. > export OMP_NUM_THREADS=... > mpiexec ... python script.py > > or: > > from firedrake import * > op2.init(backend='openmp') > ... > > Note that our current experience is that the OpenMP backend does not offer many (if any) performance improvements over just MPI mode, if you're solving big elliptic problems using AMG, you may wish to use it due to decreased memory pressure. > > > CUDA only works for a subset of firedrake functionality. In particular, the following currently don't work: > > - Solving mixed systems > - Nonlinear solves > - Extruded meshes > - Solving linear systems with CUDA + MPI > > Furthermore, for moderately complicated forms, our current strategy for parallelisation on GPUs is far from optimal (to the point where the compiler can often fail), work is starting in this area, but it's only in early stages. > > Florian has been doing some recent benchmarking in this area, so perhaps he will comment in more detail. > > >/ Also, does anyone have experience on using Intel MIC coprocessors with Firedrake? > / > I believe Fabio has done some work in this region, but I'm not sure if he did it running the full toolchain, perhaps he can comment. > > Cheers, > > Lawrence

3 5

Fwd: [Icfd-announce] Numerical Methods for Simulation - Call for Papers
by Colin Cotter 22 Sep '14

22 Sep '14

Dear all, This is usually a nice event. all the best --cjc -------- Forwarded Message -------- Subject: [Icfd-announce] Numerical Methods for Simulation - Call for Papers Date: Mon, 22 Sep 2014 09:22:32 +0000 From: Peter K. Sweby <p.k.sweby(a)reading.ac.uk> To: icfd-announce(a)lists.reading.ac.uk <icfd-announce(a)lists.reading.ac.uk> Dear all, Please find below and attached the announcement for the IMA Conference on Numerical Methods for Simulation to take place in September 2015. Unfortunately the ICFD will not be continuing their own conference series but this IMA series encapsulates the original vision of the ICFD conference series whilst expanding it to cover other areas of numerical simulation. I hope that you will support this series as you did the ICFD conferences. Best regards Peter Sweby */IMA_RGB_logo/**//* *IMA Conference on Numerical Methods for Simulation* /Formerly the ICFD Conference on Numerical Methods for Fluid Dynamics/ 1 – 4 September 2015 Mathematical Institute, University of Oxford ** *Scope* Developments in numerical methods underpin simulations in many ways, for example in any area where high dimensional problems are governed by differential equations. Computational fluid dynamics has driven many developments in this area; however there is a wide range of application areas where the problems, and indeed solution techniques may be similar. Numerical methods are important in diverse areas such as geophysical modelling, fluid-structure interaction, high-dimensional dynamical systems, weather prediction, climate modelling, oil reservoir simulation, and so on. The conference will bring together application specialists, applied mathematicians, numerical analysts and computational scientists who develop and use numerical simulations. Applications which focus on data assimilation, inverse problems, uncertainties or control, which contain as a major component a high-dimensional forward model, will also be represented. The Institute for Computational Fluid Dynamics, founded in 1983 by Bill Morton and Mike Baines, previously ran a successful sequence of international conferences based on themes of relevance to fluid dynamics. The IMA conference series reprises this original vision, but opens up a wider range of themes which are similar to those in fluid dynamics, now including all areas of numerical simulation. *Conference themes* ** For the first IMA Conference on Numerical Methods in Simulation, the themes will be: * Higher-order discretization methods * Methods for uncertainty quantification * Multi-physics / multi-material / multi-scale simulation * High performance computing (including energy efficient algorithms and algorithms for emerging architectures) * Optimization / control / design / inverse problems * Numerical analysis / numerical linear algebra and preconditioning ** *Submission of Abstracts* Abstracts are requested for contributed talks in any area consistent with the conference themes. Talks will be accepted for either oral or poster presentation on the basis of a one-page extended abstract in PDF format, which should be sent via email to conferences(a)ima.org.uk <mailto:conferences@ima.org.uk> by Friday 27 March 2015. All oral presentations will be in plenary sessions. Applications are especially encouraged for poster presentations, and these will be given higher prominence than at other similar conferences. *Proceedings* No formal proceedings will be published, although extended abstracts will be distributed to all participants. Contributors will retain copyright and are encouraged to submit their work to journals, perhaps with an acknowledgement that the paper was presented and worked on at the 2015 IMA Conference on Numerical Methods for Simulation. ** *Organising committee* Ken Morgan, Swansea University (Chair) Andy Barlow, AWE Paul Childs, Schlumberger Gould Research Dimitris Drikakis, Cranfield University Patrick Farrell, University of Oxford Peter Sweby, University of Reading Rob Scheichl, University of Bath *Further information* ** For further information, please see the conference webpage: http://ima.org.uk/conferences/conferences_calendar/numerical_methods_for_si… ** For queries or to register your interest in this event, please contact Lizzi Lake, Conference Officer, E-mail: conferences(a)ima.org.uk <mailto:conferences@ima.org.uk>, Tel: +44 (0) 1702 354 020, Fax: +44 (0) 1702 354 111, Institute of Mathematics and its Applications, Catherine Richards House, 16 Nelson Street, Southend-on-Sea, Essex, SS1 1EF, UK.

1 0

Valedictory seminar by Florian Rathgeber 1600 30 September Grantham Boardroom
by David Ham 22 Sep '14

22 Sep '14

Dear all, At the end of the month Florian is leaving us for the greener grass and tax-free climes of the European Centre for Medium-Range Weather Forecasts. Before he departs, he's going to present a seminar on the key results of his recently defended PhD. This will be an excellent opportunity to learn some more about PyOP2 and Firedrake, and how they came about. We will adjourn for leaving drinks after the seminar. Title: Solving partial differential equations efficiently and productively with Firedrake and PyOP2 - and how we got there Abstract: In this talk I will give an overview of Firedrake, a high-level framework for solving partial differential equations using the finite element method, built on top of PyOP2, a domain-specific language embedded in Python for parallel mesh-based computations. I am going to highlight characteristic features that differentiate this tool chain from existing solution, reflect on the process of designing and implementing the framework and discuss some lessons learnt in running an academic open source software project. -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

1 0

Lightweight Modular Staging
by Kelly, Paul H J 18 Sep '14

18 Sep '14

I mentioned Lightweight Modular Staging in the meeting yesterday. The context was thinking about tools to clean up our template-based code generation and allow direct construction of COFFEE or LLVM IR. LMS is done in the context of Scala so it's not directly usable - but the ideas might be helpful. Here is the website: http://scala-lms.github.io/ LMS is being used in a number of interesting DSL projects at Stanford, via a framework for domain-specific optimisations called Delite: http://ppl.stanford.edu/main/projects.html A related and somewhat similar idea has been done in Lua, as described here: http://terralang.org/pldi071-devito.pdf You can play online here http://terralang.org/demo.html Does anyone know of a similar tool in Python? As a historical note, I claim our Taskgraph Library work got there first, in C++: http://www.doc.ic.ac.uk/~phjk/Software/TGL/ Paul

2 1

Meeting this week
by Lawrence Mitchell 16 Sep '14

16 Sep '14

Today's meeting has been postponed until tomorrow, 4pm in 217 Huxley. Lawrence

3 2

Parallel in time seminar today 1200 Huxley 130
by David Ham 12 Sep '14

12 Sep '14

Greetings Firedrakers, There is a seminar today by Terry Haut who is visiting Colin and me. He's working on parallelisation, which is a subject dear to all our hearts. The twist here is that he's working on parallelising in time, which is a neat trick because you'd think that timestepping is the mother of all loop carried dependencies. The maths might be a bit heavy at times but this looks like important work. Full details at: http://www3.imperial.ac.uk/newsandeventspggrp/imperialcollege/naturalscienc… Regards, David -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

1 0

Meetings this term
by Lawrence Mitchell 08 Sep '14

08 Sep '14

Dear all, our meetings for the coming term will be in Huxley 217 on a *Tuesday* at 4pm. Exceptions: Tomorrow (9th) we're at 430 in 217. 30th September, 4th November, 18th November: location not yet determined. Cheers, Lawrence

1 0

Seminar on 12th Sept
by Colin Cotter 03 Sep '14

03 Sep '14

Dear Firedrakers, Terry Haut is giving a seminar about parallel-in-time methods for solving PDEs with fast oscillatory waves (such as those used in weather forecasting and ocean modelling). The talk will concentrate on parallelisable methods for computing the action of the exponential of an anti-symmetric matrix (i.e. the matrix coming from discretising the linear wave part of a PDE), which are then used to take large stable timesteps for the coarse propagator in a parallel-in-time algorithm. Date: 12th September 2014 Time: 12 noon Location: Huxley 130 (i.e. the maths dungeons). http://www3.imperial.ac.uk/newsandeventspggrp/imperialcollege/naturalscienc… cheers --cjc

1 0