October 2015 - firedrake - imperial.ac.uk

Re: [firedrake] weirdness on bendy branch
by Fabio Luporini 06 Oct '15

06 Oct '15

Hi Stephen, I guess this is on me. Will look into this tomorrow and will get back to you. Thanks. -- Fabio 2015-10-06 19:36 GMT+01:00 Kramer, Stephan <s.kramer(a)imperial.ac.uk>: > More weirdness, I'm afraid. If I change the function space V to: > > V = FunctionSpace(mesh2d, "RT", 1) > > I still get different answers between M and M1+M2. This is using the > bendy branches for firedrake/ffc/ufl and > enhance-rewriter-mode3-logflops for COFFEE. Again M==M1+M2 using > parameters['coffee']={} and I get the same answer for M in "flat" > firedrake. > > Any other magic branches I could try? ;-) > Seriously though, any suggestions are welcome > > Cheers > Stephan > > > On 02/10/15 13:10, Stephan Kramer wrote: > > Hey guys > > > > I've been trying to implement a simple IP viscosity scheme for DG, but > > was getting weird results, which I think I've tracked down to a bug > > related to fd_bendy. > > > > If I run the following: > > > > from firedrake import * > > > > mesh2d = UnitSquareMesh(1,1) > > V = VectorFunctionSpace(mesh2d, "DG", 1) > > uv = Function(V) > > U_test = TestFunction(V) > > > > n = FacetNormal(mesh2d) > > F1 = dot(jump(U_test[0], n), jump(uv[0], n))*dS > > F2 = -dot(avg(grad(U_test[0])), jump(uv[0], n))*dS > > F=F1+F2 > > F0 = dot(jump(U_test[0],n)-avg(grad(U_test[0])), jump(uv[0], n))*dS > > > > M=assemble(derivative(F, uv)).M.values > > M1=assemble(derivative(F1, uv)).M.values > > M2=assemble(derivative(F2, uv)).M.values > > M0=assemble(derivative(F0, uv)).M.values > > > > I really expect M, M1+M2 and M0 to be all the same. On firedrake master > > (and in fact Dolfin) these are all the same indeed. But in my fd_bendy > > enviroment (fd_bendy for ffc and ufl, bendy_changes for firedrake) the > > values for M are different. Final piece of information is that if I use > > parameters['coffee']={}, everything is fine again. Other than that I'm > > stuck though, so I'd much appreciate any pointers into what I could > > check next > > > > Cheers > > Stephan > > > > _______________________________________________ > > firedrake mailing list > > firedrake(a)imperial.ac.uk > > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > > > > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake >

1 0

Re: [firedrake] The wonderful world of firedrake-install
by David Ham 06 Oct '15

06 Oct '15

Hi Justin, On Fri, 2 Oct 2015 at 09:41 Justin Chang <jychang48(a)gmail.com> wrote: > Hi David, > > So I am using LANL’s system because I am getting tired of my university’s > cluster and how poorly it’s been managed, and had no issues with > installation. > Excellent. > > I do, however, want to bring up a few comments/questions: > > 1) Firedrake was installed on my machine with the option > -no_package_manager but I can’t update it through the update script. When I > attempt to run "/path/to/firedrake/bin/firedrake-update" it gives me an > error when it attempts to execute “apt-get”. Is there a way to update > firedrake that was installed via -no_package_manager at the moment? > This is a bug, which I will fix shortly. In the meantime, the workaround is to edit firedrake-update and replace: args.package_manager = True with args.package_manager = False > > 2) I think Lawrence mentioned this previously, but if PETSC_DIR is > detected, can you make the install script exit? Because many of us PETSc > users have our own version, and it gets really confusing/annoying when we > attempt to install firedrake and have to start over because our own PETSc > versions crash the installation. > This can be changed (when I get a moment - or when some nice person puts in a pull-request which fixes it ;). > > 3) On a related note, every time we try to reinstall firedrake with the > script, we have to remove everything and start from scratch. Is there a way > for the install script to “continue where it left off” if you get what I > mean? Because some parts i.e., PETSc take an extremely long time to build > and if we get an error because of an error towards the end of the > installation, we have to recompile PETSc each and every friggin time. > I thought about this. It's a lot of work to do in a safe way. Basically one would have to go through the whole script and reason about what state the script would be in if each step failed, then work out a teardown strategy to put the installation into a consistent state so that the next attempt has some chance of working. Once you factor in the steps you'd have to take to ensure that the system is still safe if the user runs with different command line arguments or environment variables (which they probably would because *something* must have caused the failure) then it starts to look like a major project. Accordingly, while resuming from a previously failed install would be kind of convenient, I've put it in the "too hard" basket. > > 4) Not sure if this is an issue or not, but I noticed that there are no > compiler optimization flags for the PETSc installation. If we install PETSc > —with-debugging=0, it will give us a warning that we should add additional > optimization flags like -O2 or -O3. Yes this makes the file size much > bigger especially with all the downloaded external packages, but I was > wondering if these additional optimization will actually make firedrake > faster. > Probably, yes. I'm not entirely sure what the default install does in this regard. I will defer to Lawrence's superior PETSc knowledge... > > That’s all I can think of for now. Again thank you guys for all your help, > > Justin > > On Sep 10, 2015, at 3:56 PM, Justin Chang <jychang48(a)gmail.com> wrote: > > Hi David, > > I will get back to you in a moment about this, > > Thanks, > Justin > > > On Thu, Sep 10, 2015 at 4:38 AM, David Ham <David.Ham(a)imperial.ac.uk> > wrote: > >> Hi Justin, >> >> Sorry for the delay. I'm rushing backwards and forwards between the >> University of Reading and Imperial this week so lots of balls are getting >> dropped. >> >> The hdf5 problem is yet another "pip doesn't really look after >> dependencies" issue. I think we need to forcibly move the PETSc install >> earlier in the install process (because it's PETSc which installs the >> hdf5). I'll give that a try when I'm on a real internet connection. >> >> Your problem on your cluster is that the just in time compilation of one >> of your forms is failing. If you read the error log closely, it tells you >> which files failed to compile (in this case >> it's /scratch/306103/pyop2-cache-uid3003/c47dfce75aa72266c548fad137321af9.c) >> It also tells you where the related files containing the compile log and >> compile errors are. Could you post those files please? >> >> Cheers, >> >> David >> >> On Mon, 7 Sep 2015 at 00:01 Justin Chang <jychang48(a)gmail.com> wrote: >> >>> David, >>> >>> The error is still there even with the firedrake-install >>> from install-cython-workaround. For some strange reason I cannot link >>> the mip library to Cython using pip install ... but I compile Cython 0.23 >>> from source (./configure --prefix=/home/jchang23/.local && make && make >>> install) it links properly. I suspect this may have more to do with how my >>> local cluster's path dependencies behave. >>> >>> But now, i get this other error as shown in the attached >>> firedrake-install.log. I am guessing this has to do with not having an HDF5 >>> library available within my custom installed Python? >>> >>> Also, I note from inspection that apt-get normally installs the >>> blas/lapack libraries for you, so I added the tag '--download-fblaslapack' >>> to the petsc configure options. Is that the right thing to do or should I >>> somehow install blas/lapack as a whole so that all packages depend on it? >>> >>> Finally, I finally got firedrake to install properly when I loaded the >>> intel MPI compiled python libraries. They happened to have all of the most >>> up-to-date dependencies, and installed with 'python firedrake-install >>> --user --no_package_manager --disable_ssh'. When I run my programs on the >>> head node (and avoid getting yelled at), the RT0 code I have attached runs >>> fine, but when I submit a batch script as shown in runCompare, i get the >>> outputs as shown in output_error/test. Can you guys dissect what's going on >>> or would this be more of an issue that needs to be discussed with the >>> system admins? >>> >>> Thanks, >>> Justin >>> >>> >>> On Fri, Sep 4, 2015 at 4:14 AM, David Ham <David.Ham(a)imperial.ac.uk> >>> wrote: >>> >>>> Hi Justin, >>>> >>>> The prospective fix is in install-cython-workaround . Can you try that >>>> and tell me if it works for you? It will be merged to master as soon as >>>> Travis gives the green light. >>>> >>>> Cheers, >>>> >>>> David >>>> >>>> On Fri, 4 Sep 2015 at 09:42 Ham, David A <david.ham(a)imperial.ac.uk> >>>> wrote: >>>> >>>>> Hi Justin, >>>>> >>>>> Ah yes. I saw something like this when attempting to set up travis. >>>>> The problem appears to be that the dependency analysis in pip is broken so >>>>> it doesn't realise it has to build and install Cython before it builds >>>>> h5py. I think the workaround is to force Cython installation in the install >>>>> script before looking at the requirements file. I will push a branch >>>>> presently and merge it as soon as it passes tests. >>>>> >>>>> Cheers, >>>>> >>>>> David >>>>> >>>>> On Fri, 4 Sep 2015 at 08:17 Justin Chang <jychang48(a)gmail.com> wrote: >>>>> >>>>>> Hi David >>>>>> >>>>>> Thanks for the updated script. So I am using the system provided >>>>>> openmpi/1.8.3 module but built my own Python 2.7.10 and pip from source. I >>>>>> am getting an error as shown in the attached firedrake-install.log >>>>>> >>>>>> I also got a similar issue when I built my own openmpi library, >>>>>> although in that case I didn’t have the cython related errors and only had >>>>>> the ones related to hdf5/h5py >>>>>> >>>>>> Any idea what this is? >>>>>> >>>>>> Thanks, >>>>>> Justin >>>>>> >>>>>> >>>>>> On Sep 3, 2015, at 8:54 AM, David Ham <David.Ham(a)imperial.ac.uk> >>>>>> wrote: >>>>>> >>>>>> The branch is merged now so I believe this is now fixed. >>>>>> >>>>>> David >>>>>> >>>>>> On Thu, 3 Sep 2015 at 12:09 David Ham <David.Ham(a)imperial.ac.uk> >>>>>> wrote: >>>>>> >>>>>> It turns out that this is for the usual reason: I am an idiot. A fix >>>>>> for this is in the further-install-fixes branch and I will merge the branch >>>>>> as soon as tests pass. >>>>>> >>>>>> (What's actually going on is that I am abusing CalledProcessError >>>>>> when I should really raise a special exception for this case. I have now >>>>>> created a new exception and raised that instead). >>>>>> >>>>>> Cheers, >>>>>> >>>>>> David >>>>>> >>>>>> On Thu, 3 Sep 2015 at 01:56 Justin Chang <jychang48(a)gmail.com> wrote: >>>>>> >>>>>> Thanks David, >>>>>> >>>>>> I actually forgot about the --no_package_manager option. But when I >>>>>> tried installing with it, i get this error: >>>>>> >>>>>> $ python firedrake-install --no_package_manager >>>>>> Traceback (most recent call last): >>>>>> File "firedrake-install", line 265, in <module> >>>>>> raise subprocess.CalledProcessError >>>>>> TypeError: __init__() takes at least 3 arguments (1 given) >>>>>> >>>>>> Even if I add the --disable_ssh and --developer tags I get the above >>>>>> error. Am I suppose to do something else with this? >>>>>> >>>>>> Thanks, >>>>>> Justin >>>>>> >>>>>> On Tue, Sep 1, 2015 at 3:57 AM, David Ham <David.Ham(a)imperial.ac.uk> >>>>>> wrote: >>>>>> >>>>>> Thanks, >>>>>> >>>>>> On Sun, 30 Aug 2015 at 08:15 Justin Chang <jychang48(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>> I agree that hype and a sparse direct solver like mumps is necessary. >>>>>> The latter needs --download-metis --download-parmetis >>>>>> --download-scalapack. >>>>>> >>>>>> >>>>>> Added and currently testing. >>>>>> >>>>>> >>>>>> More comments: >>>>>> >>>>>> 4) If mumps is to be included, the install script has to ensure that >>>>>> one has cmake >=2.5 when configuring metis, otherwise PETSc configure >>>>>> will return an error. Perhaps include "brew install cmake" and "sudo >>>>>> apt-get install cmake" in the script? >>>>>> >>>>>> >>>>>> Also added. Thanks for catching this. >>>>>> >>>>>> >>>>>> 5) Should --download-exodusii (and it's dependencies --download-netcdf >>>>>> --download-hdf5) also be included in the PETSC_CONFIGURE_OPTIONS? I >>>>>> imagine that if one wants to solve problems on large-scale >>>>>> unstructured grids, .exo files would be more feasible to use over .msh >>>>>> files. Or is exodusii already accounted for in firedrake/PyOP2/etc? >>>>>> >>>>>> >>>>>> And this. >>>>>> >>>>>> >>>>>> 6) If I run the script on a pristine MacOSX, I get an error saying >>>>>> "OSError: Unable to find SWIG installation. Please install SWIG >>>>>> version 2.0.0 or higher." I am guessing "brew install swig" was >>>>>> somehow missed in the firedrake-install script? >>>>>> >>>>>> Ah yes. Fecking swig. I've added the dependency to the new branch. >>>>>> Hopefully the swig dependency is going away very soon. >>>>>> >>>>>> >>>>>> >>>>>> 7) Perhaps not as important, but do y'all think it's possible to make >>>>>> this script more non-Ubuntu HPC system friendly? For isntance, either >>>>>> a) require the user to install his or her own local OpenMPI and Python >>>>>> libraries from source, b) let script download the tarballs/git >>>>>> repositories remotely and install it for you (kind of like how PETSc >>>>>> handles external packages through --download-<package>), or c) enable >>>>>> the user to point to the system provided OpenMPI and Python libraries. >>>>>> Same would need to be done for CMake, SWIG, and PCRE as these packages >>>>>> cannot simply be obtained from pip. >>>>>> >>>>>> >>>>>> I think (c) is the case right now. If you run the script on a >>>>>> non-Ubuntu system or pass the --no_package_manager option then it will just >>>>>> assume that you have installed the dependencies somewhere visible and will >>>>>> get on with pip. >>>>>> >>>>>> I think (b) is [a] hard and [b] in the case of MPI probably a bad >>>>>> idea. Supercomputers are such a diverse bunch of hacked up distributions >>>>>> that I think that installing the compiled dependencies automatically in a >>>>>> way which is portable would require a lot of install script work. In the >>>>>> case of MPI, you really need to use the MPI build which talks to the fast >>>>>> interconnect on your supercomputer, so automatically building vanilla >>>>>> OpenMPI is a Bad Thing. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> David >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Justin >>>>>> >>>>>> On Fri, Aug 28, 2015 at 7:20 AM, Lawrence Mitchell >>>>>> <lawrence.mitchell(a)imperial.ac.uk> wrote: >>>>>> > >>>>>> >> On 28 Aug 2015, at 14:01, David Ham <David.Ham(a)imperial.ac.uk> >>>>>> wrote: >>>>>> >> >>>>>> >> Hi All, >>>>>> >> >>>>>> >> I have a branch going through testing now which will handle >>>>>> PETSC_CONFIGURE_OPTIONS in a smarter way. Basically we'll just make sure >>>>>> that the things which are required are there, and we'll also honour >>>>>> anything the user has already set. I've also updated docs to say that >>>>>> that's what happens. >>>>>> >> >>>>>> >> I am also open to the suggestion that we could add more >>>>>> configuration options to the default set. Would anyone like to suggest what >>>>>> they should be? >>>>>> > >>>>>> > I think --download-hypre=1 at least. Maybe also a sparse direct >>>>>> solver (mumps?) which I think needs --download-metis --download-parmetis >>>>>> --download-mumps (maybe some others?) >>>>>> > >>>>>> > Lawrence >>>>>> > >>>>>> > >>>>>> > _______________________________________________ >>>>>> > firedrake mailing list >>>>>> > firedrake(a)imperial.ac.uk >>>>>> > https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>>>>> > >>>>>> >>>>>> _______________________________________________ >>>>>> firedrake mailing list >>>>>> firedrake(a)imperial.ac.uk >>>>>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> firedrake mailing list >>>>>> firedrake(a)imperial.ac.uk >>>>>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> firedrake mailing list >>>>>> firedrake(a)imperial.ac.uk >>>>>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>>>>> >>>>>> >>>>>> >>>> _______________________________________________ >>>> firedrake mailing list >>>> firedrake(a)imperial.ac.uk >>>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>>> >>>> >>> >> _______________________________________________ >> firedrake mailing list >> firedrake(a)imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >> >> >

1 0

Re: [firedrake] JacobianDeterminant
by Andrew McRae 04 Oct '15

04 Oct '15

Gyaaaaaaaah! On 2 October 2015 at 00:36, Tuomas Karna <tuomas.karna(a)gmail.com> wrote: > FYI: > > > abs(ufl.JacobianDeterminant(mesh.ufl_domain()))**(1.0/mesh.cell_dimension()) > > doesn't work on extruded mesh as mesh.cell_dimension() is a tuple. > > > > On 10/01/2015 04:34 PM, Tuomas Karna wrote: > > Thanks Andrew, very helpful. > > We're trying to figure out if CellSize could be generalized for extruded > meshes to return the horizontal/vertical size of the mesh. Presumably one > could just use the same Jacobian formulation to do that (e.g. det of upper > 2x2 corner for horizontal, lower left element for vertical size, assuming > extrusion is along z axis)? > > > On 10/01/2015 01:30 PM, Andrew McRae wrote: > > To give some background information, in FEniCS and in Firedrake master, > CellSize gives the circumdiameter (twice the circumradius) of the cell. > This is not implemented for quads and extruded meshes in Firedrake (I > hope...), since the circumradius isn't defined in general. > > In the Firedrake bendy_changes branch, I tentatively changed CellSize to > return (abs(JacobianDeterminant))**(1/dim) since this generalises better to > non-simplicies, higher-order elements, etc. However, this is a poor idea > since it changes the behaviour of existing [simplex] code that depends > sensitively on CellSize giving the cell circumdiameter. > > > Perhaps one solution would be to rename that CellSize to something else to > avoid the conflict? In that case we could then include the > horizontal/vertical option in the same function. > > - Tuomas > > > > On 1 October 2015 at 21:13, Andrew McRae <A.T.T.McRae(a)bath.ac.uk> wrote: > >> So you're using the 'bendy' branch in Firedrake as well? >> >> If so, JacobianDeterminant gets torn apart into components of grad(x) [x >> = coordinate field] within UFL. >> >> ufl/algorithms/change_to_reference.py: >> >> @memoized_handler >> def jacobian_determinant(self, o): >> domain = o.domain() >> J = self.jacobian(Jacobian(domain)) >> return determinant_expr(J) >> >> >> where determinant_expr is the obvious thing (use git grep >> determinant_expr from within a UFL subdirectory if you want to look at this) >> >> Note there's some magic that maps JacobianDeterminant to the handler >> jacobian_determinant. >> >> I think that pointing CellSize at JacobianDeterminant within Firedrake is >> something I'll revert in the future, since it's not backwards compatible >> for simplices (despite generalising well to other shapes). Will need to >> find another name, or just let people use the UFL object, like for >> FacetNormal. >> >> >> On 1 October 2015 at 21:01, Tuomas Karna < <tuomas.karna(a)gmail.com> >> tuomas.karna(a)gmail.com> wrote: >> >>> Hi all, >>> >>> We're trying to figure out what CellSize actually does. >>> Does anyone know where JacobianDeterminant gets turn into code? >>> >>> - Tuomas >>> >>> >>> _______________________________________________ >>> firedrake mailing list >>> firedrake(a)imperial.ac.uk >>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>> >> >> > > > _______________________________________________ > firedrake mailing listfiredrake@imperial.ac.ukhttps://mailman.ic.ac.uk/mailman/listinfo/firedrake > > > >

1 0

Re: [firedrake] Submitting a firedrake job script
by Andrew McRae 03 Oct '15

03 Oct '15

Ugh, it doesn't look like it. >From utility_meshes.py, _cachedir = os.path.join(tempfile.gettempdir(), 'firedrake-mesh-cache-uid%d' % os.getuid()) compared to, in ffc_interface.py, if MPI.comm.rank == 0: _cachedir = environ.get('FIREDRAKE_FFC_KERNEL_CACHE_DIR', path.join(tempfile.gettempdir(), 'firedrake-ffc-kernel-cache-uid%d' % getuid())) Maybe the obvious change will just work? Or maybe something subtle needs to be done in parallel, I don't know. On 3 October 2015 at 12:39, Justin Chang <jychang48(a)gmail.com> wrote: > Ah yes that's exactly what I was missing: those environment variables. Is > there one for firedrake-mesh-cache-* ? Because export > FIREDRAKE_MESH_CACHE_DIR=... did not work and resorted back to /tmp > > The issue 2) remains, though it's not as much of a concern now if i > "preprocess" the cache on a single compute node > > On Sat, Oct 3, 2015 at 4:13 AM, Andrew McRae <A.T.T.McRae(a)bath.ac.uk> > wrote: > >> You probably want something like the following in your job submission >> script: >> >> PYOP2_CACHE_DIR=$WORK/pyop2-cache >> export PYOP2_CACHE_DIR >> >> FIREDRAKE_FFC_KERNEL_CACHE_DIR=$WORK/firedrake-ffc-cache >> export FIREDRAKE_FFC_KERNEL_CACHE_DIR >> >> where $WORK is visible from all nodes. Otherwise I think the default >> location for these folders is /tmp, which is probably local to each node. >> >> Your issue 2) seems to be related to using firedrake with prefork; >> Lawrence can say more here. I think the 'solution' is to revert some >> Firedrake commits :-\ >> >> >> On 3 October 2015 at 10:21, Justin Chang <jychang48(a)gmail.com> wrote: >> >>> Hi all, >>> >>> In another thread I had a long running issue with firedrake on an HPC >>> machine. I *think* I found part of the problem. >>> >>> Everytime I submit a job script to a compute node, it has to (re)compile >>> all the FFC/OP2 forms. What I did to circumvent this problem was modify the >>> job script so that I run my firedrake program twice: once with a very small >>> mesh (to compile all the necessities) and again to simulate my actual >>> finite element problem. In the first run, I compiled the code on a single >>> MPI processes, so if the subsequent run is performed on that same node, I >>> have no issues. However, if it has to run on two different compute nodes, >>> my program freezes because I suspect that the ranks on the original node >>> has the cache/compiled forms whereas the other node does not, hence my >>> program hanging. >>> >>> All that said, I have two questions: >>> >>> 1) If I already have the compiled/cached forms of my firedrake program, >>> how do I make it so that when I submit a job, the new compute node does not >>> need to recompile my program? >>> >>> 2) Attached is an output summary of what happens when the both runs are >>> across two compute nodes (#MSUB -l nodes=2:ppn=1). Can you guys dissect >>> what's going on from this? >>> >>> Thanks, >>> Justin >>> >> >> >> _______________________________________________ >> firedrake mailing list >> firedrake(a)imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >> >> >

1 0

Re: [firedrake] Submitting a firedrake job script
by Andrew McRae 03 Oct '15

03 Oct '15

You probably want something like the following in your job submission script: PYOP2_CACHE_DIR=$WORK/pyop2-cache export PYOP2_CACHE_DIR FIREDRAKE_FFC_KERNEL_CACHE_DIR=$WORK/firedrake-ffc-cache export FIREDRAKE_FFC_KERNEL_CACHE_DIR where $WORK is visible from all nodes. Otherwise I think the default location for these folders is /tmp, which is probably local to each node. Your issue 2) seems to be related to using firedrake with prefork; Lawrence can say more here. I think the 'solution' is to revert some Firedrake commits :-\ On 3 October 2015 at 10:21, Justin Chang <jychang48(a)gmail.com> wrote: > Hi all, > > In another thread I had a long running issue with firedrake on an HPC > machine. I *think* I found part of the problem. > > Everytime I submit a job script to a compute node, it has to (re)compile > all the FFC/OP2 forms. What I did to circumvent this problem was modify the > job script so that I run my firedrake program twice: once with a very small > mesh (to compile all the necessities) and again to simulate my actual > finite element problem. In the first run, I compiled the code on a single > MPI processes, so if the subsequent run is performed on that same node, I > have no issues. However, if it has to run on two different compute nodes, > my program freezes because I suspect that the ranks on the original node > has the cache/compiled forms whereas the other node does not, hence my > program hanging. > > All that said, I have two questions: > > 1) If I already have the compiled/cached forms of my firedrake program, > how do I make it so that when I submit a job, the new compute node does not > need to recompile my program? > > 2) Attached is an output summary of what happens when the both runs are > across two compute nodes (#MSUB -l nodes=2:ppn=1). Can you guys dissect > what's going on from this? > > Thanks, > Justin >

2 1

Submitting a firedrake job script
by Justin Chang 03 Oct '15

03 Oct '15

Hi all, In another thread I had a long running issue with firedrake on an HPC machine. I *think* I found part of the problem. Everytime I submit a job script to a compute node, it has to (re)compile all the FFC/OP2 forms. What I did to circumvent this problem was modify the job script so that I run my firedrake program twice: once with a very small mesh (to compile all the necessities) and again to simulate my actual finite element problem. In the first run, I compiled the code on a single MPI processes, so if the subsequent run is performed on that same node, I have no issues. However, if it has to run on two different compute nodes, my program freezes because I suspect that the ranks on the original node has the cache/compiled forms whereas the other node does not, hence my program hanging. All that said, I have two questions: 1) If I already have the compiled/cached forms of my firedrake program, how do I make it so that when I submit a job, the new compute node does not need to recompile my program? 2) Attached is an output summary of what happens when the both runs are across two compute nodes (#MSUB -l nodes=2:ppn=1). Can you guys dissect what's going on from this? Thanks, Justin

1 0

Re: [firedrake] Flux calculation
by Colin Cotter 03 Oct '15

03 Oct '15

Yes, that's right, we are currently using a reference state in H1. The Met Office uses the previous state, but I'm wondering what is currently done in Dynamo. I went back to my notes after a discussion with Nigel and found that I never addressed this particular issue so I have emailed him again. all the best --cjc On 3 October 2015 at 10:13, Eike Mueller <E.Mueller(a)bath.ac.uk> wrote: > Hi Colin, > > thanks. Yes, I can see how one would want to use either DG or FV for this, > since the field \rho is discontinuous, and hence \rho*u is also > discontinuous. If I assume for a moment that \rho = 1, then I should just > be able to just construct the weak form of the operator, which is > <\psi,\div(u)>, and this becomes psi*div(u)*dx (with \psi being a test > function). I think what is happening in the code is that for \rho*u an > approximation F \in Hdiv is constructed (this requires the evaluation of > \rho at the nodes of u), and then this is used to construct psi*div(F)*dx > via an integral over the cell. > > Does that make sense? > > The Met Office uses the previous state, as far as I can tell. How would a > fixed reference state help? Is it because you can then express \rho as a > function in H1? > > Eike > > Sent from my iPad > > On 3 Oct 2015, at 09:44, Colin Cotter <colin.cotter(a)imperial.ac.uk> wrote: > > Ah, I see! (you need a + not a - by the way). > > That would come from a DG discretisation, or finite volume at lowest > order. The only difference between DG and finite volume is that in FV the > volume integral vanishes, whilst the surface flux integrals remain (which > would be constructed in the way that David described in Firedrake). > > all the best > --cjc > > On 3 October 2015 at 09:23, Eike Mueller <E.Mueller(a)bath.ac.uk> wrote: > >> Hi Colin, >> >> it's coming from the implicit discretisation of the mass conservation >> equation: >> >> d\rho/dt - \div(\rho*u) = 0. >> >> Thanks, >> >> Eike >> >> Sent from my iPad >> >> On 3 Oct 2015, at 09:07, Colin Cotter <colin.cotter(a)imperial.ac.uk> >> wrote: >> >> Hi Eike, >> It still very much depends on what equation is being solved in what >> space, and how it has been linearised. If you tell us which part of the >> model it is from then I can tell you how we do it in our codes. >> All the best >> Cjc >> On 3 Oct 2015 08:09, "Eike Mueller" <E.Mueller(a)bath.ac.uk> wrote: >> >>> Hi David, >>> >>> I'm trying to understand some code in DYNAMO, where it is done as I >>> described (at least that's what I think - I'm reverse engineering the >>> existing code, and waiting for clarification from the Met Office). I was >>> hoping to shed some light on this by finding out how fluxes are done in >>> firedrake. >>> >>> Ultimately, I want to assemble the weak form of the linear operator >>> >>> u -> A(u) = \div F with F=\phi*u >>> >>> i.e. <\psi, \div(\phi*u)> with u \in Hdiv, \psi, \phi \in L2. \psi \in >>> L2 is a test function. \phi is a background reference field, and in a sense >>> it doesn't really matter how the flux is constructed, as long as \phi*u is >>> a good approximation to the continuum expression (this is because the >>> linear operator shows up in the linearisation of the non-linear problem). >>> The action of A is implemented by constructing F as described and then >>> looping over all cells and calculating the divergence as <\theta,\div F>. >>> Would a more natural definition of the flux be to find F such that >>> >>> phi*dot(u,v)*dx = dot(F,v)*dx >>> >>> for all test for functions v? The thing is, I want to really assemble a >>> linear operator A, so having a solve in there wouldn't work. >>> >>> Thanks, >>> >>> Eike >>> >>> Sent from my iPad >>> >>> On 3 Oct 2015, at 04:36, David Ham <David.Ham(a)imperial.ac.uk> wrote: >>> >>> Hi Eike, >>> >>> Can you give some context. What you have described is a very >>> unfiniteelement concept. The usual FE concept of the flux is the integral >>> over a facet: >>> >>> phi*dot(u, n) * ds >>> >>> Where n is the FacetNormal. In that case everything gets evaluated at >>> quadratic equation points so the issue you raised does not occur. >>> >>> Cheers, >>> >>> David >>> >>> On Fri, 2 Oct 2015 at 20:21 Eike Mueller <E.Mueller(a)bath.ac.uk> wrote: >>> >>>> Dear firedrakers >>>> >>>> if I have a scalar field phi and a vector field u, how can I calculate >>>> the flux f=phi*u in firedrake? I need to be able to loop over the dofs i of >>>> u, and for each one calculate f_i = sum_k phi_k*e_k(x_i)*u_i where e_k(x_i) >>>> is the k-th basis function of phi evaluated at the node x_i of u. But I >>>> somehow have to tell it that I want phi evaluated at u's nodes, not the >>>> other way round. >>>> >>>> Thanks, >>>> >>>> Eike >>>> >>>> Sent from my iPad >>>> _______________________________________________ >>>> firedrake mailing list >>>> firedrake(a)imperial.ac.uk >>>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>>> >>> _______________________________________________ >>> firedrake mailing list >>> firedrake(a)imperial.ac.uk >>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>> >>> _______________________________________________ >> firedrake mailing list >> firedrake(a)imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >> >> > > > -- > http://www.imperial.ac.uk/people/colin.cotter > > www.cambridge.org/9781107663916 > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > -- http://www.imperial.ac.uk/people/colin.cotter www.cambridge.org/9781107663916

1 0

Re: [firedrake] Flux calculation
by Colin Cotter 03 Oct '15

03 Oct '15

Ah, I see! (you need a + not a - by the way). That would come from a DG discretisation, or finite volume at lowest order. The only difference between DG and finite volume is that in FV the volume integral vanishes, whilst the surface flux integrals remain (which would be constructed in the way that David described in Firedrake). all the best --cjc On 3 October 2015 at 09:23, Eike Mueller <E.Mueller(a)bath.ac.uk> wrote: > Hi Colin, > > it's coming from the implicit discretisation of the mass conservation > equation: > > d\rho/dt - \div(\rho*u) = 0. > > Thanks, > > Eike > > Sent from my iPad > > On 3 Oct 2015, at 09:07, Colin Cotter <colin.cotter(a)imperial.ac.uk> wrote: > > Hi Eike, > It still very much depends on what equation is being solved in what > space, and how it has been linearised. If you tell us which part of the > model it is from then I can tell you how we do it in our codes. > All the best > Cjc > On 3 Oct 2015 08:09, "Eike Mueller" <E.Mueller(a)bath.ac.uk> wrote: > >> Hi David, >> >> I'm trying to understand some code in DYNAMO, where it is done as I >> described (at least that's what I think - I'm reverse engineering the >> existing code, and waiting for clarification from the Met Office). I was >> hoping to shed some light on this by finding out how fluxes are done in >> firedrake. >> >> Ultimately, I want to assemble the weak form of the linear operator >> >> u -> A(u) = \div F with F=\phi*u >> >> i.e. <\psi, \div(\phi*u)> with u \in Hdiv, \psi, \phi \in L2. \psi \in L2 >> is a test function. \phi is a background reference field, and in a sense it >> doesn't really matter how the flux is constructed, as long as \phi*u is a >> good approximation to the continuum expression (this is because the linear >> operator shows up in the linearisation of the non-linear problem). The >> action of A is implemented by constructing F as described and then looping >> over all cells and calculating the divergence as <\theta,\div F>. Would a >> more natural definition of the flux be to find F such that >> >> phi*dot(u,v)*dx = dot(F,v)*dx >> >> for all test for functions v? The thing is, I want to really assemble a >> linear operator A, so having a solve in there wouldn't work. >> >> Thanks, >> >> Eike >> >> Sent from my iPad >> >> On 3 Oct 2015, at 04:36, David Ham <David.Ham(a)imperial.ac.uk> wrote: >> >> Hi Eike, >> >> Can you give some context. What you have described is a very >> unfiniteelement concept. The usual FE concept of the flux is the integral >> over a facet: >> >> phi*dot(u, n) * ds >> >> Where n is the FacetNormal. In that case everything gets evaluated at >> quadratic equation points so the issue you raised does not occur. >> >> Cheers, >> >> David >> >> On Fri, 2 Oct 2015 at 20:21 Eike Mueller <E.Mueller(a)bath.ac.uk> wrote: >> >>> Dear firedrakers >>> >>> if I have a scalar field phi and a vector field u, how can I calculate >>> the flux f=phi*u in firedrake? I need to be able to loop over the dofs i of >>> u, and for each one calculate f_i = sum_k phi_k*e_k(x_i)*u_i where e_k(x_i) >>> is the k-th basis function of phi evaluated at the node x_i of u. But I >>> somehow have to tell it that I want phi evaluated at u's nodes, not the >>> other way round. >>> >>> Thanks, >>> >>> Eike >>> >>> Sent from my iPad >>> _______________________________________________ >>> firedrake mailing list >>> firedrake(a)imperial.ac.uk >>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>> >> _______________________________________________ >> firedrake mailing list >> firedrake(a)imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >> >> _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > -- http://www.imperial.ac.uk/people/colin.cotter www.cambridge.org/9781107663916

2 2

Re: [firedrake] Flux calculation
by Colin Cotter 03 Oct '15

03 Oct '15

Hi Eike, It still very much depends on what equation is being solved in what space, and how it has been linearised. If you tell us which part of the model it is from then I can tell you how we do it in our codes. All the best Cjc On 3 Oct 2015 08:09, "Eike Mueller" <E.Mueller(a)bath.ac.uk> wrote: > Hi David, > > I'm trying to understand some code in DYNAMO, where it is done as I > described (at least that's what I think - I'm reverse engineering the > existing code, and waiting for clarification from the Met Office). I was > hoping to shed some light on this by finding out how fluxes are done in > firedrake. > > Ultimately, I want to assemble the weak form of the linear operator > > u -> A(u) = \div F with F=\phi*u > > i.e. <\psi, \div(\phi*u)> with u \in Hdiv, \psi, \phi \in L2. \psi \in L2 > is a test function. \phi is a background reference field, and in a sense it > doesn't really matter how the flux is constructed, as long as \phi*u is a > good approximation to the continuum expression (this is because the linear > operator shows up in the linearisation of the non-linear problem). The > action of A is implemented by constructing F as described and then looping > over all cells and calculating the divergence as <\theta,\div F>. Would a > more natural definition of the flux be to find F such that > > phi*dot(u,v)*dx = dot(F,v)*dx > > for all test for functions v? The thing is, I want to really assemble a > linear operator A, so having a solve in there wouldn't work. > > Thanks, > > Eike > > Sent from my iPad > > On 3 Oct 2015, at 04:36, David Ham <David.Ham(a)imperial.ac.uk> wrote: > > Hi Eike, > > Can you give some context. What you have described is a very > unfiniteelement concept. The usual FE concept of the flux is the integral > over a facet: > > phi*dot(u, n) * ds > > Where n is the FacetNormal. In that case everything gets evaluated at > quadratic equation points so the issue you raised does not occur. > > Cheers, > > David > > On Fri, 2 Oct 2015 at 20:21 Eike Mueller <E.Mueller(a)bath.ac.uk> wrote: > >> Dear firedrakers >> >> if I have a scalar field phi and a vector field u, how can I calculate >> the flux f=phi*u in firedrake? I need to be able to loop over the dofs i of >> u, and for each one calculate f_i = sum_k phi_k*e_k(x_i)*u_i where e_k(x_i) >> is the k-th basis function of phi evaluated at the node x_i of u. But I >> somehow have to tell it that I want phi evaluated at u's nodes, not the >> other way round. >> >> Thanks, >> >> Eike >> >> Sent from my iPad >> _______________________________________________ >> firedrake mailing list >> firedrake(a)imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >> > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > >

2 1

Re: [firedrake] Flux calculation
by David Ham 03 Oct '15

03 Oct '15

Hi Eike, Can you give some context. What you have described is a very unfiniteelement concept. The usual FE concept of the flux is the integral over a facet: phi*dot(u, n) * ds Where n is the FacetNormal. In that case everything gets evaluated at quadratic equation points so the issue you raised does not occur. Cheers, David On Fri, 2 Oct 2015 at 20:21 Eike Mueller <E.Mueller(a)bath.ac.uk> wrote: > Dear firedrakers > > if I have a scalar field phi and a vector field u, how can I calculate the > flux f=phi*u in firedrake? I need to be able to loop over the dofs i of u, > and for each one calculate f_i = sum_k phi_k*e_k(x_i)*u_i where e_k(x_i) is > the k-th basis function of phi evaluated at the node x_i of u. But I > somehow have to tell it that I want phi evaluated at u's nodes, not the > other way round. > > Thanks, > > Eike > > Sent from my iPad > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake >

2 1