August 2014 - firedrake - imperial.ac.uk

Re: [firedrake] Firedrake on supercomputers
by Tuomas Karna 24 Sep '14

24 Sep '14

Hi Lawrence, Thanks for these, this is great. On stampede I can access the compilers in the devel queue so this works like a charm. Would you have any hints for improving parallel scalabilty? Right now things start to level out at 8 cores (2d shallow water with p2-p1 elems, 76k triangles). Also, is there another tool for timings than PYOP2_PRINT_SUMMARY? Cheers, Tuomas > Hi Tuomas, > > comments in line below. > > On 7 Aug 2014, at 07:31, Tuomas Karna <tuomas.karna at gmail.com <https://mailman.ic.ac.uk/mailman/listinfo/firedrake>> wrote: > > >/ Hi all, > />/ > />/ I'm running my code on TACC Stampede and I have a couple of questions: > />/ > />/ How can I setup Firedrake/PyOP2 for the target machine, for example to use Intel compilers, target architecture (sandybridge) instruction set, or Intel MKL libraries? > / > Note that for all this to work, you need to be able to launch the compiler on compute nodes (in particular for intel, the compute nodes will have to be able to talk to the license server), but I hope this all works. > > Selection of the intel compiler is: > > from firedrake import * > parameters["coffee"]["compiler"] = "intel" > > This still launches 'mpicc' but uses intel specific flags when compiling. If the compiler is actually called something different (e.g. on Cray systems it's always cc) then you need to do: > > export CC=name_of_cc_compiler > # only do this if the linker is separate from the compiler > export LDSHARED=name_of_linker > > and then: > > from firedrake import * > parameters["coffee"]["compiler"] = "intel" > > In addition, you probably want to select the AVX instruction set: > > parameters["coffee"]["simd_isa"] = "avx" > > Finally, you can select various code transformations to be applied to the generated kernels to increase their performance (seehttp://arxiv.org/abs/1407.0904 for many details). In particular you may wish to try: > > # loop invariant code motion > parameters["coffee"]["licm"] = True > > # Data alignment (to give compilers a better chance at vectorising) > parameters["coffee"]["ap"] = True > > # Request the compiler try harder to auto-vectorise > from pyop2.coffee.ast_plan import AUTOVECT > parameters["coffee"]["vect"] = (AUTOVECT, -1) > > > Our experience is that for most forms, the big gains come from loop invariant code motion and data alignment, but Fabio can comment in much more detail on this. > > Finally, usage of MKL, by default, none of the code Firedrake/PyOP2 generate uses BLAS at all, so if you want to exploit BLAS in the solver library, you just needed to compile PETSc appropriately. You can apply code transformations to the kernels to convert them into BLAS calls, but for low order, it may not be much of a win (since we do not amortise the call-overhead across elements), but you can try it: > > # BLAS-transformations aren't exposed through the parameters dict so: > from firedrake import * > op2.configuration["blas"] = "mkl" > > ... > > You can also, if it's available, try Eigen instead: > > op2.configuration["blas"] = "eigen" > > > >/ Until now I've only used MPI. What do I need to do to run with openmp or cuda for instance? I've got the PyOP2 dependencies in place, as they are listed on the Firedrake website. > / > Running with openmp requires that you've built PETSc "-with-threadcomm --with-openmp --with-pthreadclasses". You can then either do: > > export PYOP2_BACKEND=openmp > # Or however your batch system wants you to do this. > export OMP_NUM_THREADS=... > mpiexec ... python script.py > > or: > > from firedrake import * > op2.init(backend='openmp') > ... > > Note that our current experience is that the OpenMP backend does not offer many (if any) performance improvements over just MPI mode, if you're solving big elliptic problems using AMG, you may wish to use it due to decreased memory pressure. > > > CUDA only works for a subset of firedrake functionality. In particular, the following currently don't work: > > - Solving mixed systems > - Nonlinear solves > - Extruded meshes > - Solving linear systems with CUDA + MPI > > Furthermore, for moderately complicated forms, our current strategy for parallelisation on GPUs is far from optimal (to the point where the compiler can often fail), work is starting in this area, but it's only in early stages. > > Florian has been doing some recent benchmarking in this area, so perhaps he will comment in more detail. > > >/ Also, does anyone have experience on using Intel MIC coprocessors with Firedrake? > / > I believe Fabio has done some work in this region, but I'm not sure if he did it running the full toolchain, perhaps he can comment. > > Cheers, > > Lawrence

3 5

Normal petsc4py service has now resumed.
by David Ham 28 Aug '14

28 Aug '14

Hi All, Lisandro just merged my fixes for the pestc interface changes so you can go back to using petsc4py master in accordance with the instructions on the website. Cheers, David -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

1 0

Petsc4py installation
by David Ham 26 Aug '14

26 Aug '14

Dear all, It seems our friends at PETSc have changed the PETSc interface without a corresponding petsc4py change. Again. I have a branch which fixes this. I will be putting in a pull request imminently. In the meantime, if you upgrade PETSc to current master, you will need to use my petsc4py branch. This can be installed (system-wide) with: sudo pip install -U --no-deps git+ https://bitbucket.org/David_Ham/petsc4py.git@DMDASetDim_interface_change#eg… I will let you know once normal service resumes. David -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

2 1

Re: [firedrake] petsc4py install issues
by Andrew McRae 25 Aug '14

25 Aug '14

That question doesn't make sense, because a Git branch is no more than a named commit (every commit knows its entire history, by virtue of pointing to the parent commit[s]). However, there's bound to be some Git command that asks 'does commit X appear in the history of commit Y?'. More practically, you could look at the Firedrake branches page on Github: https://github.com/firedrakeproject/firedrake/branches. This shows that the master branch built successfully on Travis 2 days ago, and this implies there was no petsc/petsc4py mismatch at that moment. Armed with this information, it's easy to narrow down the relevant merge. In what way is the problem not fixed? Have you cleaned petsc/petsc4py? (sudo make allclean / sudo make fullclean) On 24 August 2014 20:05, Cotter, Colin J <colin.cotter(a)imperial.ac.uk> wrote: > Thanks. Is it possible to work out which branch c73cfb54 is on, just by > looking at the BitBucket page? I was having trouble tracking it back > because the branch commit appears on a different page from the merge. > --cjc > ------------------------------ > *From:* firedrake-bounces(a)imperial.ac.uk [firedrake-bounces(a)imperial.ac.uk] > on behalf of Andrew McRae [a.mcrae12(a)imperial.ac.uk] > *Sent:* 24 August 2014 19:55 > > *To:* firedrake > *Subject:* Re: [firedrake] petsc4py install issues > > 22d5acb seems like a more obvious choice? (the revision before Matt > Knepley merged his pull request, according to > https://bitbucket.org/petsc/petsc/commits/branch/master ) > > > On 24 August 2014 19:50, Cotter, Colin J <colin.cotter(a)imperial.ac.uk> > wrote: > >> (by didn't work, I mean that it didn't fix the problem for me). >> >> --cjc >> ------------------------------ >> *From:* firedrake-bounces(a)imperial.ac.uk [ >> firedrake-bounces(a)imperial.ac.uk] on behalf of Cotter, Colin J [ >> colin.cotter(a)imperial.ac.uk] >> *Sent:* 24 August 2014 19:49 >> >> *To:* firedrake >> *Subject:* Re: [firedrake] petsc4py install issues >> >> OK, the first revision I tried didn't work (df3898e >> <https://bitbucket.org/petsc/petsc/commits/df3898ee999f8e28aa22e84b9bfe0d47e…>), >> now trying (8a23364 >> <https://bitbucket.org/petsc/petsc/commits/8a23364d02173c6cd1856e280835b29b1…> >> ). >> >> --cjc >> ------------------------------ >> *From:* firedrake-bounces(a)imperial.ac.uk [ >> firedrake-bounces(a)imperial.ac.uk] on behalf of Cotter, Colin J [ >> colin.cotter(a)imperial.ac.uk] >> *Sent:* 24 August 2014 19:40 >> *To:* firedrake >> *Subject:* Re: [firedrake] petsc4py install issues >> >> OK, I think I've done that and it's building. >> >> I'm not feeling the separation of concerns here... >> >> --cjc >> ------------------------------ >> *From:* firedrake-bounces(a)imperial.ac.uk [ >> firedrake-bounces(a)imperial.ac.uk] on behalf of Andrew McRae [ >> a.mcrae12(a)imperial.ac.uk] >> *Sent:* 24 August 2014 19:24 >> *To:* firedrake >> *Subject:* Re: [firedrake] petsc4py install issues >> >> Try reverting to a PETSc commit predating the commit that Florian >> isolated. >> >> >> On 24 August 2014 19:18, Cotter, Colin J <colin.cotter(a)imperial.ac.uk> >> wrote: >> >>> Oh I see, it's an incompatibility between master Petsc and master >>> petsc4py. I wouldn't have a clue how to fix that. >>> Cjc >>> On 24 Aug 2014 19:00, "David Ham" <David.Ham(a)imperial.ac.uk> wrote: >>> >>>> We don't have our own PETSc versions. We're tracking master. The PETSc >>>> crowd muck up fairly frequently. It sounds like you should be able to fix >>>> Petsc4py pretty easily. They usually accept patches for this sort of thing >>>> pretty fast. >>>> >>>> David >>>> >>>> On Sunday, August 24, 2014, Cotter, Colin J < >>>> colin.cotter(a)imperial.ac.uk> wrote: >>>> >>>>> Aren't the Firedrake petsc and petsc4py versions supposed to be >>>>> compatible with each other? >>>>> On 24 Aug 2014 16:58, "Florian Rathgeber" < >>>>> f.rathgeber10(a)imperial.ac.uk> wrote: >>>>> >>>>>> The usual: interface change in PETSc (DMDASetDim renamed to >>>>>> DMDASetDimension) that hasn't been updated in petsc4py yet. The PETSc >>>>>> revision introducing that change is c73cfb54. >>>>>> >>>>>> On 24/08/14 16:27, Cotter, Colin J wrote: >>>>>> > Dear Firedrakers, >>>>>> > I just updated my petsc and petsc4py using the PyOP2 install >>>>>> > instructions, to be precise I did: >>>>>> > >>>>>> >> sudo PETSC_CONFIGURE_OPTIONS="--download-ctetgen >>>>>> --download-triangle >>>>>> > --download-chaco" pip install -U --no-deps >>>>>> > git+https://bitbucket.org/petsc/petsc.git#egg=petsc >>>>>> >> unset PETSC_DIR >>>>>> >> unset PETSC_ARCH >>>>>> >> sudo pip install -U --no-deps git+ >>>>>> https://bitbucket.org/petsc/petsc4py.git >>>>>> > >>>>>> > I then updated and built PyOP2, firedrake too. >>>>>> > >>>>>> > Now, upon importing Firedrake, I get: >>>>>> > >>>>>> > In [1]: from firedrake import * >>>>>> > >>>>>> --------------------------------------------------------------------------- >>>>>> > ImportError Traceback (most recent >>>>>> call last) >>>>>> > /home/cjc1/mercurial/shapefem/<ipython-input-1-379626444471> in >>>>>> <module>() >>>>>> > ----> 1 from firedrake import * >>>>>> > >>>>>> > /home/cjc1/firedrake/firedrake/firedrake/__init__.py in <module>() >>>>>> > 1 # Ensure petsc is initialised by us before anything else >>>>>> gets in >>>>>> > there. >>>>>> > >>>>>> > ----> 2 import petsc >>>>>> > 3 del petsc >>>>>> > 4 >>>>>> > 5 from ufl import * >>>>>> > >>>>>> > /home/cjc1/firedrake/firedrake/firedrake/petsc.py in <module>() >>>>>> > 2 import petsc4py >>>>>> > 3 import sys >>>>>> > ----> 4 petsc4py.init(sys.argv) >>>>>> > 5 from petsc4py import PETSc # NOQA get flake8 to ignore >>>>>> unused >>>>>> > import. >>>>>> > >>>>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/__init__.pyc in >>>>>> > init(args, arch, comm) >>>>>> > 40 """ >>>>>> > 41 import petsc4py.lib >>>>>> > ---> 42 PETSc = petsc4py.lib.ImportPETSc(arch) >>>>>> > 43 args = petsc4py.lib.getInitArgs(args) >>>>>> > 44 PETSc._initialize(args, comm) >>>>>> > >>>>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/__init__.pyc in >>>>>> > ImportPETSc(arch) >>>>>> > 27 """ >>>>>> > 28 path, arch = getPathArchPETSc(arch) >>>>>> > ---> 29 return Import('petsc4py', 'PETSc', path, arch) >>>>>> > 30 >>>>>> > 31 def getPathArchPETSc(arch=None): >>>>>> > >>>>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/__init__.pyc in >>>>>> > Import(pkg, name, path, arch) >>>>>> > 62 pathlist = [os.path.join(path, arch)] >>>>>> > 63 fo, fn, stuff = imp.find_module(name, pathlist) >>>>>> > ---> 64 module = imp.load_module(fullname, fo, fn, stuff) >>>>>> > 65 module.__arch__ = arch # save arch value >>>>>> > 66 setattr(sys.modules[pkg], name, module) >>>>>> > >>>>>> > ImportError: >>>>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/PETSc.so: >>>>>> undefined >>>>>> > symbol: DMDASetDim >>>>>> > >>>>>> > What's wrong here? >>>>>> > >>>>>> > all the best >>>>>> > --Colin >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> firedrake mailing list >>>>>> firedrake(a)imperial.ac.uk >>>>>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>>>>> >>>>>> >>>> >>>> -- >>>> Dr David Ham >>>> Departments of Mathematics and Computing >>>> Imperial College London >>>> >>>> http://www.imperial.ac.uk/people/david.ham >>>> >>>> >>>> _______________________________________________ >>>> firedrake mailing list >>>> firedrake(a)imperial.ac.uk >>>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>>> >>>> >> >

2 1

Re: [firedrake] petsc4py install issues
by Andrew McRae 24 Aug '14

24 Aug '14

22d5acb seems like a more obvious choice? (the revision before Matt Knepley merged his pull request, according to https://bitbucket.org/petsc/petsc/commits/branch/master ) On 24 August 2014 19:50, Cotter, Colin J <colin.cotter(a)imperial.ac.uk> wrote: > (by didn't work, I mean that it didn't fix the problem for me). > > --cjc > ------------------------------ > *From:* firedrake-bounces(a)imperial.ac.uk [firedrake-bounces(a)imperial.ac.uk] > on behalf of Cotter, Colin J [colin.cotter(a)imperial.ac.uk] > *Sent:* 24 August 2014 19:49 > > *To:* firedrake > *Subject:* Re: [firedrake] petsc4py install issues > > OK, the first revision I tried didn't work (df3898e > <https://bitbucket.org/petsc/petsc/commits/df3898ee999f8e28aa22e84b9bfe0d47e…>), > now trying (8a23364 > <https://bitbucket.org/petsc/petsc/commits/8a23364d02173c6cd1856e280835b29b1…> > ). > > --cjc > ------------------------------ > *From:* firedrake-bounces(a)imperial.ac.uk [firedrake-bounces(a)imperial.ac.uk] > on behalf of Cotter, Colin J [colin.cotter(a)imperial.ac.uk] > *Sent:* 24 August 2014 19:40 > *To:* firedrake > *Subject:* Re: [firedrake] petsc4py install issues > > OK, I think I've done that and it's building. > > I'm not feeling the separation of concerns here... > > --cjc > ------------------------------ > *From:* firedrake-bounces(a)imperial.ac.uk [firedrake-bounces(a)imperial.ac.uk] > on behalf of Andrew McRae [a.mcrae12(a)imperial.ac.uk] > *Sent:* 24 August 2014 19:24 > *To:* firedrake > *Subject:* Re: [firedrake] petsc4py install issues > > Try reverting to a PETSc commit predating the commit that Florian > isolated. > > > On 24 August 2014 19:18, Cotter, Colin J <colin.cotter(a)imperial.ac.uk> > wrote: > >> Oh I see, it's an incompatibility between master Petsc and master >> petsc4py. I wouldn't have a clue how to fix that. >> Cjc >> On 24 Aug 2014 19:00, "David Ham" <David.Ham(a)imperial.ac.uk> wrote: >> >>> We don't have our own PETSc versions. We're tracking master. The PETSc >>> crowd muck up fairly frequently. It sounds like you should be able to fix >>> Petsc4py pretty easily. They usually accept patches for this sort of thing >>> pretty fast. >>> >>> David >>> >>> On Sunday, August 24, 2014, Cotter, Colin J <colin.cotter(a)imperial.ac.uk> >>> wrote: >>> >>>> Aren't the Firedrake petsc and petsc4py versions supposed to be >>>> compatible with each other? >>>> On 24 Aug 2014 16:58, "Florian Rathgeber" <f.rathgeber10(a)imperial.ac.uk> >>>> wrote: >>>> >>>>> The usual: interface change in PETSc (DMDASetDim renamed to >>>>> DMDASetDimension) that hasn't been updated in petsc4py yet. The PETSc >>>>> revision introducing that change is c73cfb54. >>>>> >>>>> On 24/08/14 16:27, Cotter, Colin J wrote: >>>>> > Dear Firedrakers, >>>>> > I just updated my petsc and petsc4py using the PyOP2 install >>>>> > instructions, to be precise I did: >>>>> > >>>>> >> sudo PETSC_CONFIGURE_OPTIONS="--download-ctetgen --download-triangle >>>>> > --download-chaco" pip install -U --no-deps >>>>> > git+https://bitbucket.org/petsc/petsc.git#egg=petsc >>>>> >> unset PETSC_DIR >>>>> >> unset PETSC_ARCH >>>>> >> sudo pip install -U --no-deps git+ >>>>> https://bitbucket.org/petsc/petsc4py.git >>>>> > >>>>> > I then updated and built PyOP2, firedrake too. >>>>> > >>>>> > Now, upon importing Firedrake, I get: >>>>> > >>>>> > In [1]: from firedrake import * >>>>> > >>>>> --------------------------------------------------------------------------- >>>>> > ImportError Traceback (most recent >>>>> call last) >>>>> > /home/cjc1/mercurial/shapefem/<ipython-input-1-379626444471> in >>>>> <module>() >>>>> > ----> 1 from firedrake import * >>>>> > >>>>> > /home/cjc1/firedrake/firedrake/firedrake/__init__.py in <module>() >>>>> > 1 # Ensure petsc is initialised by us before anything else >>>>> gets in >>>>> > there. >>>>> > >>>>> > ----> 2 import petsc >>>>> > 3 del petsc >>>>> > 4 >>>>> > 5 from ufl import * >>>>> > >>>>> > /home/cjc1/firedrake/firedrake/firedrake/petsc.py in <module>() >>>>> > 2 import petsc4py >>>>> > 3 import sys >>>>> > ----> 4 petsc4py.init(sys.argv) >>>>> > 5 from petsc4py import PETSc # NOQA get flake8 to ignore >>>>> unused >>>>> > import. >>>>> > >>>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/__init__.pyc in >>>>> > init(args, arch, comm) >>>>> > 40 """ >>>>> > 41 import petsc4py.lib >>>>> > ---> 42 PETSc = petsc4py.lib.ImportPETSc(arch) >>>>> > 43 args = petsc4py.lib.getInitArgs(args) >>>>> > 44 PETSc._initialize(args, comm) >>>>> > >>>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/__init__.pyc in >>>>> > ImportPETSc(arch) >>>>> > 27 """ >>>>> > 28 path, arch = getPathArchPETSc(arch) >>>>> > ---> 29 return Import('petsc4py', 'PETSc', path, arch) >>>>> > 30 >>>>> > 31 def getPathArchPETSc(arch=None): >>>>> > >>>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/__init__.pyc in >>>>> > Import(pkg, name, path, arch) >>>>> > 62 pathlist = [os.path.join(path, arch)] >>>>> > 63 fo, fn, stuff = imp.find_module(name, pathlist) >>>>> > ---> 64 module = imp.load_module(fullname, fo, fn, stuff) >>>>> > 65 module.__arch__ = arch # save arch value >>>>> > 66 setattr(sys.modules[pkg], name, module) >>>>> > >>>>> > ImportError: >>>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/PETSc.so: >>>>> undefined >>>>> > symbol: DMDASetDim >>>>> > >>>>> > What's wrong here? >>>>> > >>>>> > all the best >>>>> > --Colin >>>>> >>>>> >>>>> _______________________________________________ >>>>> firedrake mailing list >>>>> firedrake(a)imperial.ac.uk >>>>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>>>> >>>>> >>> >>> -- >>> Dr David Ham >>> Departments of Mathematics and Computing >>> Imperial College London >>> >>> http://www.imperial.ac.uk/people/david.ham >>> >>> >>> _______________________________________________ >>> firedrake mailing list >>> firedrake(a)imperial.ac.uk >>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>> >>> >

2 2

Re: [firedrake] petsc4py install issues
by Andrew McRae 24 Aug '14

24 Aug '14

Try reverting to a PETSc commit predating the commit that Florian isolated. On 24 August 2014 19:18, Cotter, Colin J <colin.cotter(a)imperial.ac.uk> wrote: > Oh I see, it's an incompatibility between master Petsc and master > petsc4py. I wouldn't have a clue how to fix that. > Cjc > On 24 Aug 2014 19:00, "David Ham" <David.Ham(a)imperial.ac.uk> wrote: > >> We don't have our own PETSc versions. We're tracking master. The PETSc >> crowd muck up fairly frequently. It sounds like you should be able to fix >> Petsc4py pretty easily. They usually accept patches for this sort of thing >> pretty fast. >> >> David >> >> On Sunday, August 24, 2014, Cotter, Colin J <colin.cotter(a)imperial.ac.uk> >> wrote: >> >>> Aren't the Firedrake petsc and petsc4py versions supposed to be >>> compatible with each other? >>> On 24 Aug 2014 16:58, "Florian Rathgeber" <f.rathgeber10(a)imperial.ac.uk> >>> wrote: >>> >>>> The usual: interface change in PETSc (DMDASetDim renamed to >>>> DMDASetDimension) that hasn't been updated in petsc4py yet. The PETSc >>>> revision introducing that change is c73cfb54. >>>> >>>> On 24/08/14 16:27, Cotter, Colin J wrote: >>>> > Dear Firedrakers, >>>> > I just updated my petsc and petsc4py using the PyOP2 install >>>> > instructions, to be precise I did: >>>> > >>>> >> sudo PETSC_CONFIGURE_OPTIONS="--download-ctetgen --download-triangle >>>> > --download-chaco" pip install -U --no-deps >>>> > git+https://bitbucket.org/petsc/petsc.git#egg=petsc >>>> >> unset PETSC_DIR >>>> >> unset PETSC_ARCH >>>> >> sudo pip install -U --no-deps git+ >>>> https://bitbucket.org/petsc/petsc4py.git >>>> > >>>> > I then updated and built PyOP2, firedrake too. >>>> > >>>> > Now, upon importing Firedrake, I get: >>>> > >>>> > In [1]: from firedrake import * >>>> > >>>> --------------------------------------------------------------------------- >>>> > ImportError Traceback (most recent call >>>> last) >>>> > /home/cjc1/mercurial/shapefem/<ipython-input-1-379626444471> in >>>> <module>() >>>> > ----> 1 from firedrake import * >>>> > >>>> > /home/cjc1/firedrake/firedrake/firedrake/__init__.py in <module>() >>>> > 1 # Ensure petsc is initialised by us before anything else gets >>>> in >>>> > there. >>>> > >>>> > ----> 2 import petsc >>>> > 3 del petsc >>>> > 4 >>>> > 5 from ufl import * >>>> > >>>> > /home/cjc1/firedrake/firedrake/firedrake/petsc.py in <module>() >>>> > 2 import petsc4py >>>> > 3 import sys >>>> > ----> 4 petsc4py.init(sys.argv) >>>> > 5 from petsc4py import PETSc # NOQA get flake8 to ignore unused >>>> > import. >>>> > >>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/__init__.pyc in >>>> > init(args, arch, comm) >>>> > 40 """ >>>> > 41 import petsc4py.lib >>>> > ---> 42 PETSc = petsc4py.lib.ImportPETSc(arch) >>>> > 43 args = petsc4py.lib.getInitArgs(args) >>>> > 44 PETSc._initialize(args, comm) >>>> > >>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/__init__.pyc in >>>> > ImportPETSc(arch) >>>> > 27 """ >>>> > 28 path, arch = getPathArchPETSc(arch) >>>> > ---> 29 return Import('petsc4py', 'PETSc', path, arch) >>>> > 30 >>>> > 31 def getPathArchPETSc(arch=None): >>>> > >>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/__init__.pyc in >>>> > Import(pkg, name, path, arch) >>>> > 62 pathlist = [os.path.join(path, arch)] >>>> > 63 fo, fn, stuff = imp.find_module(name, pathlist) >>>> > ---> 64 module = imp.load_module(fullname, fo, fn, stuff) >>>> > 65 module.__arch__ = arch # save arch value >>>> > 66 setattr(sys.modules[pkg], name, module) >>>> > >>>> > ImportError: >>>> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/PETSc.so: >>>> undefined >>>> > symbol: DMDASetDim >>>> > >>>> > What's wrong here? >>>> > >>>> > all the best >>>> > --Colin >>>> >>>> >>>> _______________________________________________ >>>> firedrake mailing list >>>> firedrake(a)imperial.ac.uk >>>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>>> >>>> >> >> -- >> Dr David Ham >> Departments of Mathematics and Computing >> Imperial College London >> >> http://www.imperial.ac.uk/people/david.ham >> >> >> _______________________________________________ >> firedrake mailing list >> firedrake(a)imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >> >>

2 3

Re: [firedrake] petsc4py install issues
by David Ham 24 Aug '14

24 Aug '14

We don't have our own PETSc versions. We're tracking master. The PETSc crowd muck up fairly frequently. It sounds like you should be able to fix Petsc4py pretty easily. They usually accept patches for this sort of thing pretty fast. David On Sunday, August 24, 2014, Cotter, Colin J <colin.cotter(a)imperial.ac.uk> wrote: > Aren't the Firedrake petsc and petsc4py versions supposed to be > compatible with each other? > On 24 Aug 2014 16:58, "Florian Rathgeber" <f.rathgeber10(a)imperial.ac.uk > <javascript:_e(%7B%7D,'cvml','f.rathgeber10(a)imperial.ac.uk');>> wrote: > >> The usual: interface change in PETSc (DMDASetDim renamed to >> DMDASetDimension) that hasn't been updated in petsc4py yet. The PETSc >> revision introducing that change is c73cfb54. >> >> On 24/08/14 16:27, Cotter, Colin J wrote: >> > Dear Firedrakers, >> > I just updated my petsc and petsc4py using the PyOP2 install >> > instructions, to be precise I did: >> > >> >> sudo PETSC_CONFIGURE_OPTIONS="--download-ctetgen --download-triangle >> > --download-chaco" pip install -U --no-deps >> > git+https://bitbucket.org/petsc/petsc.git#egg=petsc >> >> unset PETSC_DIR >> >> unset PETSC_ARCH >> >> sudo pip install -U --no-deps git+ >> https://bitbucket.org/petsc/petsc4py.git >> > >> > I then updated and built PyOP2, firedrake too. >> > >> > Now, upon importing Firedrake, I get: >> > >> > In [1]: from firedrake import * >> > >> --------------------------------------------------------------------------- >> > ImportError Traceback (most recent call >> last) >> > /home/cjc1/mercurial/shapefem/<ipython-input-1-379626444471> in >> <module>() >> > ----> 1 from firedrake import * >> > >> > /home/cjc1/firedrake/firedrake/firedrake/__init__.py in <module>() >> > 1 # Ensure petsc is initialised by us before anything else gets in >> > there. >> > >> > ----> 2 import petsc >> > 3 del petsc >> > 4 >> > 5 from ufl import * >> > >> > /home/cjc1/firedrake/firedrake/firedrake/petsc.py in <module>() >> > 2 import petsc4py >> > 3 import sys >> > ----> 4 petsc4py.init(sys.argv) >> > 5 from petsc4py import PETSc # NOQA get flake8 to ignore unused >> > import. >> > >> > /usr/local/lib/python2.7/dist-packages/petsc4py/__init__.pyc in >> > init(args, arch, comm) >> > 40 """ >> > 41 import petsc4py.lib >> > ---> 42 PETSc = petsc4py.lib.ImportPETSc(arch) >> > 43 args = petsc4py.lib.getInitArgs(args) >> > 44 PETSc._initialize(args, comm) >> > >> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/__init__.pyc in >> > ImportPETSc(arch) >> > 27 """ >> > 28 path, arch = getPathArchPETSc(arch) >> > ---> 29 return Import('petsc4py', 'PETSc', path, arch) >> > 30 >> > 31 def getPathArchPETSc(arch=None): >> > >> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/__init__.pyc in >> > Import(pkg, name, path, arch) >> > 62 pathlist = [os.path.join(path, arch)] >> > 63 fo, fn, stuff = imp.find_module(name, pathlist) >> > ---> 64 module = imp.load_module(fullname, fo, fn, stuff) >> > 65 module.__arch__ = arch # save arch value >> > 66 setattr(sys.modules[pkg], name, module) >> > >> > ImportError: >> > /usr/local/lib/python2.7/dist-packages/petsc4py/lib/PETSc.so: undefined >> > symbol: DMDASetDim >> > >> > What's wrong here? >> > >> > all the best >> > --Colin >> >> >> _______________________________________________ >> firedrake mailing list >> firedrake(a)imperial.ac.uk >> <javascript:_e(%7B%7D,'cvml','firedrake(a)imperial.ac.uk');> >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >> >> -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

2 1

petsc4py install issues
by Cotter, Colin J 24 Aug '14

24 Aug '14

Dear Firedrakers, I just updated my petsc and petsc4py using the PyOP2 install instructions, to be precise I did: > sudo PETSC_CONFIGURE_OPTIONS="--download-ctetgen --download-triangle --download-chaco" pip install -U --no-deps git+https://bitbucket.org/petsc/petsc.git#egg=petsc > unset PETSC_DIR > unset PETSC_ARCH > sudo pip install -U --no-deps git+https://bitbucket.org/petsc/petsc4py.git I then updated and built PyOP2, firedrake too. Now, upon importing Firedrake, I get: In [1]: from firedrake import * --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /home/cjc1/mercurial/shapefem/<ipython-input-1-379626444471> in <module>() ----> 1 from firedrake import * /home/cjc1/firedrake/firedrake/firedrake/__init__.py in <module>() 1 # Ensure petsc is initialised by us before anything else gets in there. ----> 2 import petsc 3 del petsc 4 5 from ufl import * /home/cjc1/firedrake/firedrake/firedrake/petsc.py in <module>() 2 import petsc4py 3 import sys ----> 4 petsc4py.init(sys.argv) 5 from petsc4py import PETSc # NOQA get flake8 to ignore unused import. /usr/local/lib/python2.7/dist-packages/petsc4py/__init__.pyc in init(args, arch, comm) 40 """ 41 import petsc4py.lib ---> 42 PETSc = petsc4py.lib.ImportPETSc(arch) 43 args = petsc4py.lib.getInitArgs(args) 44 PETSc._initialize(args, comm) /usr/local/lib/python2.7/dist-packages/petsc4py/lib/__init__.pyc in ImportPETSc(arch) 27 """ 28 path, arch = getPathArchPETSc(arch) ---> 29 return Import('petsc4py', 'PETSc', path, arch) 30 31 def getPathArchPETSc(arch=None): /usr/local/lib/python2.7/dist-packages/petsc4py/lib/__init__.pyc in Import(pkg, name, path, arch) 62 pathlist = [os.path.join(path, arch)] 63 fo, fn, stuff = imp.find_module(name, pathlist) ---> 64 module = imp.load_module(fullname, fo, fn, stuff) 65 module.__arch__ = arch # save arch value 66 setattr(sys.modules[pkg], name, module) ImportError: /usr/local/lib/python2.7/dist-packages/petsc4py/lib/PETSc.so: undefined symbol: DMDASetDim What's wrong here? all the best --Colin

3 2

SymPy
by Andrew McRae 15 Aug '14

15 Aug '14

Upstream FIAT introduced a dependency on SymPy. This has now been integrated into our fork of FIAT. Hence, if you update FIAT, you may find that you need to install SymPy. For example: sudo pip install sympy

1 0

Re: [firedrake] Firedrake on supercomputers
by Fabio Luporini 07 Aug '14

07 Aug '14

Hi Tuomas Comments from Lawrence are correct. For what concerns optimizing assembly through COFFEE (which is a tool integrated with PyOP2 that does the optimization of local element matrices/vectors evaluations) it is actually true that in general: parameters["coffee"]["licm"] = True parameters["coffee"]["ap"] = True are the key parameters. However, if your function spaces have relatively high polynomial order *or *if your form has a lot of coefficients, then you may be interested in other optimizations as well. Please, let me know if this is the case, in which case I'll try to be more precise. Said that, our (short-term) goal is to let you (users) abstract completely from playing with these sort of "low level parameters". Just to let you know, we are currently working on an autotuning system that, once in place, should allow you to get significant better run-times while avoiding the need for setting/trying manually individual optimizations. We are not far from that, so we'll keep you posted (should take order of days). As for the MIC, we have some performance numbers concerning assembly "in isolation", but we (I) have never brought the whole toolchain on the MIC - that is, we have just used it as an accelerator. In particular, code is *not* specifically optimized for these kind of architectures, especially if you are running at low polynomial order (in practice, we are not taking full advantage of the large vector lanes). As for MKL in assembly kernels, I'm running experiments in these days to quantify how much faster can we go by transforming assembly code into a sequence of BLAS calls. Problem is that at low polynomial order the involved matrices are kind of small. What I have seen so far (but these are early experiments) is that if you are (I'm being vague here, but just to give you an idea) using like polynomial order 3 or 4 and if you are form as some coefficients in it, then turning to BLAS may be a (big) win. I hope I'll be able to report more (convincing) results, and to be more precise, in the next few days. Hope this helps -- Fabio 2014-08-07 7:31 GMT+01:00 Tuomas Karna <tuomas.karna(a)gmail.com>: > Hi all, > > I'm running my code on TACC Stampede and I have a couple of questions: > > How can I setup Firedrake/PyOP2 for the target machine, for example to > use Intel compilers, target architecture (sandybridge) instruction set, > or Intel MKL libraries? > > Until now I've only used MPI. What do I need to do to run with openmp or > cuda for instance? I've got the PyOP2 dependencies in place, as they are > listed on the Firedrake website. > > Also, does anyone have experience on using Intel MIC coprocessors with > Firedrake? > > Cheers, > > Tuomas > > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake >

2 1