- firedrake - imperial.ac.uk

Re: [firedrake] Firedrake on supercomputers
by Fabio Luporini 07 Aug '14

07 Aug '14

Hi Tuomas Comments from Lawrence are correct. For what concerns optimizing assembly through COFFEE (which is a tool integrated with PyOP2 that does the optimization of local element matrices/vectors evaluations) it is actually true that in general: parameters["coffee"]["licm"] = True parameters["coffee"]["ap"] = True are the key parameters. However, if your function spaces have relatively high polynomial order *or *if your form has a lot of coefficients, then you may be interested in other optimizations as well. Please, let me know if this is the case, in which case I'll try to be more precise. Said that, our (short-term) goal is to let you (users) abstract completely from playing with these sort of "low level parameters". Just to let you know, we are currently working on an autotuning system that, once in place, should allow you to get significant better run-times while avoiding the need for setting/trying manually individual optimizations. We are not far from that, so we'll keep you posted (should take order of days). As for the MIC, we have some performance numbers concerning assembly "in isolation", but we (I) have never brought the whole toolchain on the MIC - that is, we have just used it as an accelerator. In particular, code is *not* specifically optimized for these kind of architectures, especially if you are running at low polynomial order (in practice, we are not taking full advantage of the large vector lanes). As for MKL in assembly kernels, I'm running experiments in these days to quantify how much faster can we go by transforming assembly code into a sequence of BLAS calls. Problem is that at low polynomial order the involved matrices are kind of small. What I have seen so far (but these are early experiments) is that if you are (I'm being vague here, but just to give you an idea) using like polynomial order 3 or 4 and if you are form as some coefficients in it, then turning to BLAS may be a (big) win. I hope I'll be able to report more (convincing) results, and to be more precise, in the next few days. Hope this helps -- Fabio 2014-08-07 7:31 GMT+01:00 Tuomas Karna <tuomas.karna(a)gmail.com>: > Hi all, > > I'm running my code on TACC Stampede and I have a couple of questions: > > How can I setup Firedrake/PyOP2 for the target machine, for example to > use Intel compilers, target architecture (sandybridge) instruction set, > or Intel MKL libraries? > > Until now I've only used MPI. What do I need to do to run with openmp or > cuda for instance? I've got the PyOP2 dependencies in place, as they are > listed on the Firedrake website. > > Also, does anyone have experience on using Intel MIC coprocessors with > Firedrake? > > Cheers, > > Tuomas > > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake >

2 1

Re: [firedrake] Promotional flyer draft.
by Fabio Luporini 07 Aug '14

07 Aug '14

I *do* like it. At a certain point you write: "The code generation approach in Firedrake enables computer scientists to advance the compiler and parallelisation technology, independent of the numerics. Numericists to develop sophisticated discretisations independent of their implementation, and application specialists to deploy these numerics efficiently and effectively." Maybe you wanted a comma before numericists and not a full stop? Good job -- Fabio 2014-08-07 16:16 GMT+01:00 Mitchell, Lawrence < lawrence.mitchell(a)imperial.ac.uk>: > > On 7 Aug 2014, at 16:02, David Ham <David.Ham(a)imperial.ac.uk> wrote: > > > Hi all, > > > > This time with pretty pictures. Any comments before I send this in most > welcome. > > Here are a few minor ones: > > H(div) and H(curl) are normally spelled with lower-case div and curl. > > It is possible to drive paraview to spit out vector graphics: > > File->Export Scene > > Select PDF output, and then untick "Rasterise 3D graphics". > > Lawrence >

1 0

Re: [firedrake] Promotional flyer draft.
by David Ham 07 Aug '14

07 Aug '14

Hi all, This time with pretty pictures. Any comments before I send this in most welcome. David On 7 August 2014 12:43, Ham, David A <david.ham(a)imperial.ac.uk> wrote: > The PRISM project (http://prism.ac.uk/) of which we are part, is > developing two page flyers about each of the software packages in the > consortium. We are producing the content, and a professional graphic > designer will lay out the actual flyers. > > The target market is the steering committee of PRISM, which is primarily > people from industry at the simulation user end of the spectrum. I think > the flyer therefore needs to be something like "what can Firedrake do for > you". > > Comments are sought on the attached text fragments. Some of us will also > be making some pretty pictures for this this afternoon. > > -- > Dr David Ham > Departments of Mathematics and Computing > Imperial College London > > http://www.imperial.ac.uk/people/david.ham > -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

2 1

Promotional flyer draft.
by David Ham 07 Aug '14

07 Aug '14

The PRISM project (http://prism.ac.uk/) of which we are part, is developing two page flyers about each of the software packages in the consortium. We are producing the content, and a professional graphic designer will lay out the actual flyers. The target market is the steering committee of PRISM, which is primarily people from industry at the simulation user end of the spectrum. I think the flyer therefore needs to be something like "what can Firedrake do for you". Comments are sought on the attached text fragments. Some of us will also be making some pretty pictures for this this afternoon. -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

1 0

Firedrake on supercomputers
by Tuomas Karna 07 Aug '14

07 Aug '14

Hi all, I'm running my code on TACC Stampede and I have a couple of questions: How can I setup Firedrake/PyOP2 for the target machine, for example to use Intel compilers, target architecture (sandybridge) instruction set, or Intel MKL libraries? Until now I've only used MPI. What do I need to do to run with openmp or cuda for instance? I've got the PyOP2 dependencies in place, as they are listed on the Firedrake website. Also, does anyone have experience on using Intel MIC coprocessors with Firedrake? Cheers, Tuomas

2 1

Numba
by Andrew McRae 03 Aug '14

03 Aug '14

Hello all, One of my badminton acquaintances asked if I'd heard of Numba <http://numba.pydata.org/> (wiki <http://en.wikipedia.org/wiki/Numba>, github <https://github.com/numba/numba>). It seems to have a role similar to that of Cython, while requiring even less intrusive changes to the code. Chances are that someone else on the list will have a better idea of whether we might be interested in using it in the future. There's a couple of blog posts here <http://jakevdp.github.io/blog/2012/08/24/numba-vs-cython/> and here <http://jakevdp.github.io/blog/2013/06/15/numba-vs-cython-take-2/> comparing Numba with Cython; they seem to have similar execution speeds so it's unlikely to be worth rewriting existing code. Andrew

1 0

Re: [firedrake] Interpolating data to Function
by Lawrence Mitchell 31 Jul '14

31 Jul '14

> On 31 Jul 2014, at 16:40, Tuomas Karna <tuomas.karna(a)gmail.com> wrote: > > Lawrence, > > Another quick question. Is there a mapping somewhere that maps the local nodes stored in func.dat.data to global indices (i.e. nodes of the entire mesh)? This does exist, but it won't do what you want, since it's not unique when repartitioning. Essentially it's just an mpi_exscan of the local numbers. The ordering of the local dofs will change on a different number of partitions, so the global ordering will be different too. In short, the dat.save/dat.load solution works when restarting on the same number of processes, but we don't have a solution at the moment for general restart, but see below. > I'd like to store the function states to disk so that I could restart simulations. func.dat.save saves the local array for each process. The partitioning however may change between runs, so I'd need to store the global array. We don't currently have a mechanism for doing this. Michael Lange has been working on incorporating the hdf5 io offered by petsc as a replacement for both visualisation output and also check pointing. However, I think it is still work in progress. Lawrence

2 1

Re: [firedrake] Interpolating data to Function
by David Ham 31 Jul '14

31 Jul '14

Hi Tuomas, The nicer Python interpolation support is now live. The documentation is at: http://www.firedrakeproject.org/firedrake.html#module-firedrake.expression Hope that's useful. Regards, David On 30 July 2014 21:45, Tuomas Karna <tuomas.karna(a)gmail.com> wrote: > > On 07/30/2014 01:34 PM, Lawrence Mitchell wrote: > >> On 30 Jul 2014, at 21:21, Tuomas Karna <tuomas.karna(a)gmail.com> wrote: > >> > >> Hi Lawrence, > >> > >>> On 07/30/2014 02:13 AM, Lawrence Mitchell wrote: > >>>> On 30/07/14 02:56, Tuomas Karna wrote: > >>>> Hi Firedrake-people, > >>>> > >>>> I'm working with simple 2D ocean model and I'd like to load > bathymetric > >>>> data to a Function, interpolating to the correct coordinates of the > >>>> nodes. Currently I get the x/y coordinates from specialized x/y > function > >>>> and then just write the data directly to Function.dat.data array. So > >>>> something like this: > >>>> > >>>> P1 = FunctionSpace(mesh, "CG", 1) > >>>> bath = Function(P1,name='bathymetry') > >>>> x_func = Function(P1).interpolate(Expression('x[0]')) > >>>> y_func = Function(P1).interpolate(Expression('x[1]')) > >>>> def interpolateBath(x,y): > >>>> # interpolate here > >>>> return 3.0 > >>>> bath.dat.data[:] = interpolateBath(x_func.dat.data, y_func.dat.data) > >>>> > >>>> Is there a cleaner way of doing this? > >>> I think there is unfortunately currently not. > >> OK, just checking if I was missing something obvious. > > Note that we have a branch in the works that allows you to define a > python class to be used in function.interpolate (this is the same as > DOLFIN's python expressions). So this may well end up doing what you want. > You'll need an updated version of pyop2 and the python_expressions > firedrake branch if you want to try it. > Sounds good, I'll take a look. > > > >>>> I have a similar question for the boundaries. I have external data > >>>> defined on a line which I'd like to interpolate to the nodes of a > >>>> certain boundary. I could use a Function for this too, if I knew the > >>>> indices of the boundary nodes. I've hacked something together using > >>>> mesh.exterior_facets and > FunctionSpace.exterior_facet_boundary_node_map, > >>>> just wondering if there is an easier way? > >>> If your mesh has a marker for that boundary, (e.g. a boundary id in > >>> Gmsh), then you can get the indices of the boundary nodes (by which I > >>> presume you mean the vertices) with: > >>> > >>> mesh = Mesh("input.msh") > >>> > >>> V = FunctionSpace(mesh, 'CG', 1) > >>> > >>> bc = DirichletBC(V, 0.0, integer_id_of_boundary) > >>> > >>> boundary_nodes = bc.nodes > >>> > >>> > >>> under the hood, this likely does something very similar to what you did > >>> with exterior_facets and the exterior_facet_boundary_node_map. > >> Thanks, I have boundary markers so bc.nodes works fine. In parallel > however it also returns indices that exceed the size of x_func.dat.data, > maybe the halo nodes are included? Is there an easy way to get the resident > (non-halo) nodes only? > > You can do one of two things: > > > > 1. Restrict the bc nodes using numpy.take. You can find out the number > of local degrees of freedom in a function in f.dof_dset.size > > > > 2. Just access the day with the halo values in place: > > > > x_func.dat.data_with_halos > > > > I think the second option is probably simpler > OK, both approaches seem to work fine. > > Thanks, > > Tuomas > > > > > Lawrence > > > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

1 0

Interpolating data to Function
by Tuomas Karna 30 Jul '14

30 Jul '14

Hi Firedrake-people, I'm working with simple 2D ocean model and I'd like to load bathymetric data to a Function, interpolating to the correct coordinates of the nodes. Currently I get the x/y coordinates from specialized x/y function and then just write the data directly to Function.dat.data array. So something like this: P1 = FunctionSpace(mesh, "CG", 1) bath = Function(P1,name='bathymetry') x_func = Function(P1).interpolate(Expression('x[0]')) y_func = Function(P1).interpolate(Expression('x[1]')) def interpolateBath(x,y): # interpolate here return 3.0 bath.dat.data[:] = interpolateBath(x_func.dat.data, y_func.dat.data) Is there a cleaner way of doing this? I have a similar question for the boundaries. I have external data defined on a line which I'd like to interpolate to the nodes of a certain boundary. I could use a Function for this too, if I knew the indices of the boundary nodes. I've hacked something together using mesh.exterior_facets and FunctionSpace.exterior_facet_boundary_node_map, just wondering if there is an easier way? Regs, Tuomas Karna Post-Doctoral Fellow Oregon Health & Science University

2 4

Re: [firedrake] Interpolating data to Function
by David Ham 30 Jul '14

30 Jul '14

Hi Tuomas, On Wednesday, July 30, 2014, Tuomas Karna <tuomas.karna(a)gmail.com> wrote: > Hi Lawrence, > > On 07/30/2014 02:13 AM, Lawrence Mitchell wrote: > > On 30/07/14 02:56, Tuomas Karna wrote: > >> Hi Firedrake-people, > >> > >> I'm working with simple 2D ocean model and I'd like to load bathymetric > >> data to a Function, interpolating to the correct coordinates of the > >> nodes. Currently I get the x/y coordinates from specialized x/y function > >> and then just write the data directly to Function.dat.data array. So > >> something like this: > >> > >> P1 = FunctionSpace(mesh, "CG", 1) > >> bath = Function(P1,name='bathymetry') > >> x_func = Function(P1).interpolate(Expression('x[0]')) > >> y_func = Function(P1).interpolate(Expression('x[1]')) > >> def interpolateBath(x,y): > >> # interpolate here > >> return 3.0 > >> bath.dat.data[:] = interpolateBath(x_func.dat.data, y_func.dat.data) > >> > >> Is there a cleaner way of doing this? > > I think there is unfortunately currently not. > OK, just checking if I was missing something obvious. This has been on the wish list for a while and you prompted us to do something about it today. Expect a merge with a better solution tomorrow our time. > >> I have a similar question for the boundaries. I have external data > >> defined on a line which I'd like to interpolate to the nodes of a > >> certain boundary. I could use a Function for this too, if I knew the > >> indices of the boundary nodes. I've hacked something together using > >> mesh.exterior_facets and FunctionSpace.exterior_facet_boundary_node_map, > >> just wondering if there is an easier way? > > If your mesh has a marker for that boundary, (e.g. a boundary id in > > Gmsh), then you can get the indices of the boundary nodes (by which I > > presume you mean the vertices) with: > > > > mesh = Mesh("input.msh") > > > > V = FunctionSpace(mesh, 'CG', 1) > > > > bc = DirichletBC(V, 0.0, integer_id_of_boundary) > > > > boundary_nodes = bc.nodes > > > > > > under the hood, this likely does something very similar to what you did > > with exterior_facets and the exterior_facet_boundary_node_map. > Thanks, I have boundary markers so bc.nodes works fine. In parallel > however it also returns indices that exceed the size of x_func.dat.data, > maybe the halo nodes are included? Is there an easy way to get the > resident (non-halo) nodes only? > > That's not straightforward, but the other way around is easy. You can just access x_func.dat.data_with_halos . It shouldn't hurt too much to do the duplicate interpolation on the halo nodes. Cheers, David > Cheers, > > Tuomas > > > Cheers, > > > > Lawrence > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk <javascript:;> > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > -- Dr David Ham Departments of Mathematics and Computing Imperial College London http://www.imperial.ac.uk/people/david.ham

1 0