March 2017 - firedrake - imperial.ac.uk

Re: [firedrake] installing firedrake on a cluster
by David Ham 05 Apr '17

05 Apr '17

Hi Francis, This is not the entire log. You've only pasted the Python exception. That Python exception was caused by something failing in PETSc's build system. However we can't see from here what that is. Please attach the whole log. Many thanks, David On Fri, 31 Mar 2017 at 13:09 Francis Poulin <fpoulin(a)uwaterloo.ca> wrote: > Thanks again. That was very helpful. > > It seemed to make it to the petsc install before it gave up. I > highlighted in read what appeared to me in red. > > Any ideas what went wrong and how I could fix it? > > Cheers, Francis > > > Traceback (most recent call last): > File "<string>", line 1, in <module> > File "/local/tmp/pip-GY2H_M-build/setup.py", line 302, in <module> > **metadata) > File > "/opt/sharcnet/python/2.7.8/intel/lib/python2.7/distutils/core.py", line > 151, in setup > dist.run_commands() > File > "/opt/sharcnet/python/2.7.8/intel/lib/python2.7/distutils/dist.py", line > 953, in run_commands > self.run_command(cmd) > File > "/opt/sharcnet/python/2.7.8/intel/lib/python2.7/distutils/dist.py", line > 972, in run_command > cmd_obj.run() > File "/local/tmp/pip-GY2H_M-build/setup.py", line 218, in run > config(prefix, self.dry_run) > File "/local/tmp/pip-GY2H_M-build/setup.py", line 148, in config > if status != 0: raise RuntimeError(status) > RuntimeError: 256 > > ---------------------------------------- > Command "/home/fpoulin/software/firedrake/bin/python -u -c "import > setuptools, > tokenize;__file__='/local/tmp/pip-GY2H_M-build/setup.py';f=getattr(tokenize, > 'open', open)(__file__);code=f.read().replace('\r\n', > '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record > /tmp/pip-fXQE7k-record/install-record.txt > --single-version-externally-managed --compile --install-headers > /home/fpoulin/software/firedrake/include/site/python2.7/petsc" failed with > error code 1 in /local/tmp/pip-GY2H_M-build/ > Traceback (most recent call last): > File "firedrake-install", line 1234, in <module> > install("petsc/") > File "firedrake-install", line 678, in install > run_pip_install(["--ignore-installed", package]) > File "firedrake-install", line 548, in run_pip_install > check_call(pipinstall + pipargs) > > File "firedrake-install", line 369, in check_call > subprocess.check_call(arguments, env=env) > File "/opt/sharcnet/python/2.7.8/intel/lib/python2.7/subprocess.py", > line 540, in check_call > raise CalledProcessError(retcode, cmd) > subprocess.CalledProcessError: Command > '['/home/fpoulin/software/firedrake/bin/pip', 'install', '--no-binary', > 'mpi4py', '--no-deps', '--ignore-installed', 'petsc/']' returned non-zero > exit status 1 > > > > > ------------------ > Francis Poulin > Associate Professor > Department of Applied Mathematics > University of Waterloo > > email: fpoulin(a)uwaterloo.ca > Web: https://uwaterloo.ca/poulin-research-group/ > Telephone: +1 519 888 4567 x32637 <(519)%20888-4567> > > ------------------------------ > *From:* firedrake-bounces(a)imperial.ac.uk [firedrake-bounces(a)imperial.ac.uk] > on behalf of David Ham [David.Ham(a)imperial.ac.uk] > *Sent:* Friday, March 31, 2017 5:55 AM > *To:* firedrake > > *Subject:* Re: [firedrake] installing firedrake on a cluster > Hi Francis and thanks Justin for the good advice! > > First, you almost certainly want to: > > unset PYTHONPATH > > instead of telling Firedrake to honour it. This is because it's unlikely > that there is something in your PYTHONPATH which you want the virtualenv to > see (this is why having PYTHONPATH set triggers this error). > > Next, assuming that you managed to install pip using Justin's instructions > (or by having the sysadmin do it), you can try running: > > pip install --user virtualenv > > yourself. If that works, try running the install script again. If it > fails, then show us the error message and we might be able to understand > what's gone wrong. > > Regards, > > David > > On Fri, 31 Mar 2017 at 03:43 Justin Chang <jychang48(a)gmail.com> wrote: > > Francis, I assume you tried get-pip.py > <https://bootstrap.pypa.io/get-pip.py>? > > If you simply run the above with system python (e.g., 'python get-pip.py') > it will return an error because it is trying to install pip into the root > directory which I am assuming you have no access to. What you need to do is > install pip locally. I did this on my local cluster through 'python > get-pip.py --user' which will install pip into $HOME/.local. I can't > remember what I did next but I think I had to ensure that this .local > directory was visible through PATH: > > export PATH=$HOME/.local:$HOME/.local/bin:$PATH > > and then pip worked. > > Justin > > On Thu, Mar 30, 2017 at 9:32 PM, Francis Poulin <fpoulin(a)uwaterloo.ca> > wrote: > > Thanks again. > > It would appear that I don't have pip installed. (see below) > > I found instructions on their site on how to do that and of course it > doesn't work for some reason. I will get to to him me set up pip and then > try again tomorrow and hope I can get a bit further. I will let you know > things progress, when they progress that is. > > Cheers, Francis > > > > [fpoulin@orc129 software]$ apt-get --version > -bash: apt-get: command not found > [fpoulin@orc129 software]$ python firedrake-install --honour-pythonpath > --no_package_manager > apt-get not found or disabled. Proceeding on the rash assumption that your > compiled dependencies are in place. > If this is not the case, please install the following and try again: > * A C and C++ compiler (for example gcc/g++ or clang), GNU make > * A Fortran compiler (for PETSc) > * MPI > * Blas and Lapack > * Git, Mercurial > * Python version 2.7 > * pip and the Python headers > * autoconf, automake, libtool > * CMake > * zlib > Installing virtualenv. > Collecting virtualenv > Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Could not find a version that satisfies the requirement virtualenv (from > versions: ) > No matching distribution found for virtualenv > Traceback (most recent call last): > File "firedrake-install", line 1172, in <module> > check_call(["pip", "install", "--user", "virtualenv"]) > File "firedrake-install", line 369, in check_call > subprocess.check_call(arguments, env=env) > File "/opt/sharcnet/python/2.7.8/intel/lib/python2.7/subprocess.py", > line 540, in check_call > raise CalledProcessError(retcode, cmd) > subprocess.CalledProcessError: Command '['pip', 'install', '--user', > 'virtualenv']' returned non-zero exit status 1 > > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > -- > Dr David Ham > Department of Mathematics > Imperial College London > -- Dr David Ham Department of Mathematics Imperial College London

2 3

DumbCheckpoint runtime error
by Pan, Wei 05 Apr '17

05 Apr '17

Hi all, When I try to store computation using DumbCheckpoint I get the following run time error: RuntimeError: Unable to convert handle to h5py object. Likely h5py not linked to same HDF5 as PETSc The full error message I get is the following Traceback (most recent call last): File "main.py", line 17, in <module> toymodel.run_euler() File "/media/data/code/MyFirstProject/toymodel.py", line 76, in run_euler run_solver(dumpfreq_coarse,coarse) File "/media/data/code/MyFirstProject/toymodel.py", line 42, in run_solver solver.euler_solver(dumpfreq, euler_params) File "/media/data/code/MyFirstProject/euler.py", line 380, in euler_solver chk = DumbCheckpoint(filename, mode = FILE_CREATE) File "/home/wpan1/Downloads/firedrake/src/firedrake/firedrake/checkpointing.py", line 65, in __init__ self.new_file() File "/home/wpan1/Downloads/firedrake/src/firedrake/firedrake/checkpointing.py", line 151, in new_file self.write_attribute("/", "nprocs", self.comm.size) File "/home/wpan1/Downloads/firedrake/src/firedrake/firedrake/checkpointing.py", line 251, in write_attribute self.h5file[obj].attrs[name] = val File "/home/wpan1/Downloads/firedrake/src/firedrake/firedrake/checkpointing.py", line 166, in h5file self._h5file = h5i.get_h5py_file(self.vwr) File "firedrake/hdf5interface.pyx", line 38, in firedrake.hdf5interface.get_h5py_file (firedrake/hdf5interface.c:1669) RuntimeError: Unable to convert handle to h5py object. Likely h5py not linked to same HDF5 as PETSc Any suggestions? Many thanks Wei

3 11

Parameter estimation and optimal experimental design with Firedrake
by Buesing, Henrik 31 Mar '17

31 Mar '17

Dear all, How would I do parameter estimation (and optimal experimental design) in Firedrake? I have a time-dependent forward model consisting of a PDE system and depending on certain parameters. I also have time-dependent measurements/data. Now, I would like to minimize some functional, which measures the distance from the forward model to the recorded data. How would I do this in Firedrake? Is there an example for this? In a next step, I would like to do optimal experimental design, where one maximizes the Fisher information (e.g. minimize the trace of the inverse of the information matrix) with respect to position. Any ideas on how to approach this? Thank you for your help! Henrik -- Dipl.-Math. Henrik Büsing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University Mathieustr. | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 http://www.eonerc.rwth-aachen.de/GGE hbuesing(a)eonerc.rwth-aachen.de

1 0

Re: [firedrake] close to fixing my nonlinear QG solver
by David Ham 31 Mar '17

31 Mar '17

Pull requests always welcome ;) On Thu, 30 Mar 2017 at 13:24 Francis Poulin <fpoulin(a)uwaterloo.ca> wrote: > I am sorry that I didn't follow the emails more closely last week. I > could have saved myself, and everyone, some time for debugging. > > When trying things out I did try help(split), thinking maybe that was the > problem. What I found is copied below. It is, let's say, terse. > > I agree with Colin, if someone could insert something on the docs spelling > out the difference that could be very helpful for other people who might > very well come across the same issue. > > Cheers, Francis > > P.S. I will go back and reread old emails to understand why this is the > correct syntax. > > > Help on function split in module ufl.split_functions: > > split(v) > UFL operator: If v is a Coefficient or Argument in a mixed space, > returns > a tuple with the function components corresponding to the subelements. > > > > > ------------------ > Francis Poulin > Associate Professor > Department of Applied Mathematics > University of Waterloo > > email: fpoulin(a)uwaterloo.ca > Web: https://uwaterloo.ca/poulin-research-group/ > Telephone: +1 519 888 4567 x32637 <(519)%20888-4567> > > ------------------------------ > *From:* firedrake-bounces(a)imperial.ac.uk [firedrake-bounces(a)imperial.ac.uk] > on behalf of Colin Cotter [colin.cotter(a)imperial.ac.uk] > *Sent:* Thursday, March 30, 2017 7:59 AM > *To:* firedrake > > *Subject:* Re: [firedrake] close to fixing my nonlinear QG solver > It's a classic gotcha. We are kind of stuck with it due to wanting to stay > close to the Fenics interface, but maybe we should have a specific bit in > the docs about it [or a more highlighted one, if it already exists]. > > all the best > --cjc > > On 30 March 2017 at 12:57, Francis Poulin <fpoulin(a)uwaterloo.ca> wrote: > > Houston, we have convergence! > > Thank you for the help Lawrence. > > I did see there were emails about the split passed around last week, which > I clearly should have read more closely. > > Now that I have convergence on a course grid I will test a finder grid and > let you know how robust this is. > > Cheers, Francis > > ------------------ > Francis Poulin > Associate Professor > Department of Applied Mathematics > University of Waterloo > > email: fpoulin(a)uwaterloo.ca > Web: https://uwaterloo.ca/poulin-research-group/ > Telephone: +1 519 888 4567 x32637 > > > ________________________________________ > From: firedrake-bounces(a)imperial.ac.uk [firedrake-bounces(a)imperial.ac.uk] > on behalf of Lawrence Mitchell [lawrence.mitchell(a)imperial.ac.uk] > Sent: Thursday, March 30, 2017 7:14 AM > To: firedrake(a)imperial.ac.uk > Subject: Re: [firedrake] close to fixing my nonlinear QG solver > > Hi Francis, > > sorry, I now saw your problem. > > You do: > > soln1.assign(soln0) > > psi1, u1, q1 = soln1.split() > > Gnon = F(psi1, u1, q1) > > You've been bitten by the "wrong split" bug!! ARGH. > > Only use function.split() for output. If you're going to use the > split values in a form, you need to use "split(function)" > > If you replace your lines: > > psi1, u1, q1 = soln1.split() > Gnon = ... > > with: > > psi1, u1, q1 = split(soln1) > Gnon = ... > psi1, u1, q1 = soln1.split() > > > Everything works. > > Lawrence > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > > > > -- > http://www.imperial.ac.uk/people/colin.cotter > > www.cambridge.org/9781107663916 > > > -- Dr David Ham Department of Mathematics Imperial College London

3 7

Re: [firedrake] installing firedrake on a cluster
by David Ham 31 Mar '17

31 Mar '17

Hi Francis and thanks Justin for the good advice! First, you almost certainly want to: unset PYTHONPATH instead of telling Firedrake to honour it. This is because it's unlikely that there is something in your PYTHONPATH which you want the virtualenv to see (this is why having PYTHONPATH set triggers this error). Next, assuming that you managed to install pip using Justin's instructions (or by having the sysadmin do it), you can try running: pip install --user virtualenv yourself. If that works, try running the install script again. If it fails, then show us the error message and we might be able to understand what's gone wrong. Regards, David On Fri, 31 Mar 2017 at 03:43 Justin Chang <jychang48(a)gmail.com> wrote: > Francis, I assume you tried get-pip.py > <https://bootstrap.pypa.io/get-pip.py>? > > If you simply run the above with system python (e.g., 'python get-pip.py') > it will return an error because it is trying to install pip into the root > directory which I am assuming you have no access to. What you need to do is > install pip locally. I did this on my local cluster through 'python > get-pip.py --user' which will install pip into $HOME/.local. I can't > remember what I did next but I think I had to ensure that this .local > directory was visible through PATH: > > export PATH=$HOME/.local:$HOME/.local/bin:$PATH > > and then pip worked. > > Justin > > On Thu, Mar 30, 2017 at 9:32 PM, Francis Poulin <fpoulin(a)uwaterloo.ca> > wrote: > > Thanks again. > > It would appear that I don't have pip installed. (see below) > > I found instructions on their site on how to do that and of course it > doesn't work for some reason. I will get to to him me set up pip and then > try again tomorrow and hope I can get a bit further. I will let you know > things progress, when they progress that is. > > Cheers, Francis > > > > [fpoulin@orc129 software]$ apt-get --version > -bash: apt-get: command not found > [fpoulin@orc129 software]$ python firedrake-install --honour-pythonpath > --no_package_manager > apt-get not found or disabled. Proceeding on the rash assumption that your > compiled dependencies are in place. > If this is not the case, please install the following and try again: > * A C and C++ compiler (for example gcc/g++ or clang), GNU make > * A Fortran compiler (for PETSc) > * MPI > * Blas and Lapack > * Git, Mercurial > * Python version 2.7 > * pip and the Python headers > * autoconf, automake, libtool > * CMake > * zlib > Installing virtualenv. > Collecting virtualenv > Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after > connection broken by 'ProtocolError('Connection aborted.', error(101, > 'Network is unreachable'))': /simple/virtualenv/ > Could not find a version that satisfies the requirement virtualenv (from > versions: ) > No matching distribution found for virtualenv > Traceback (most recent call last): > File "firedrake-install", line 1172, in <module> > check_call(["pip", "install", "--user", "virtualenv"]) > File "firedrake-install", line 369, in check_call > subprocess.check_call(arguments, env=env) > File "/opt/sharcnet/python/2.7.8/intel/lib/python2.7/subprocess.py", > line 540, in check_call > raise CalledProcessError(retcode, cmd) > subprocess.CalledProcessError: Command '['pip', 'install', '--user', > 'virtualenv']' returned non-zero exit status 1 > > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > > -- Dr David Ham Department of Mathematics Imperial College London

2 1

Fwd: [petsc-maint] Weak scaling test: Fieldsplit questions
by Matthew Knepley 31 Mar '17

31 Mar '17

What is going on here? It looks like you are using subprocess. Why would you do that on a cluster rather than MPI? Matt ---------- Forwarded message ---------- From: Buesing, Henrik <HBuesing(a)eonerc.rwth-aachen.de> Date: Thu, Mar 30, 2017 at 8:25 AM Subject: AW: [petsc-maint] Weak scaling test: Fieldsplit questions To: Matthew Knepley <knepley(a)gmail.com> Cc: Barry Smith <bsmith(a)mcs.anl.gov>, Hong <hzhang(a)mcs.anl.gov>, " petsc-maint(a)mcs.anl.gov" <petsc-maint(a)mcs.anl.gov> In my opinion, there is some kind of race condition in Firedrake when running on more than one node. Thus, until this is fixed it is very unlikely for me to get the 64 cores case running. Hmm, we are running Firedrake in parallel with no problems here. What is the error? *[Buesing, Henrik] *See [1] for the error message and the attached three logs (for the 32 core case this was a 2/5 running and 3/5 crashing ). This is just for running the compiled code. During the compile stage I had problems, too. What I did is the following: 1) Run Firedrake on 1 node (this works). Now all the *.so files are in place. 2) Run Firedrake on more than one node. This crashes more often the more processes I use. I’m guessing for a race condition, because on 17 cores (1 node + 1 core) my problem runs fine. On 32 cores it sometimes runs. And on 64 cores it, up to now, never runs. But if you are not having these problems, and if the provided code reproduces the MatCreateSubMats problem, then you can do tests on your own. Well, a lot of ifs, but better than nothing. Thank you! Henrik [1] Traceback (most recent call last): File "/work/hb111949/Firedrake/twophase/2pDrake/2pinjection.py", line 228, in <module> solver = NonlinearVariationalSolver(problem,options_prefix="") File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ src/firedrake/firedrake/variational_solver.py", line 156, in __init__ pre_function_callback=pre_f_callback) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ src/firedrake/firedrake/solving_utils.py", line 260, in __init__ form_compiler_parameters=fcp) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ src/firedrake/firedrake/assemble.py", line 143, in create_assembly_callable collect_loops=True) File "<decorator-gen-279>", line 2, in _assemble File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ src/firedrake/firedrake/utils.py", line 62, in wrapper return f(*args, **kwargs) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ src/firedrake/firedrake/assemble.py", line 192, in _assemble kernels = tsfc_interface.compile_form(f, "form", parameters=form_compiler_parameters, inverse=inverse) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ src/firedrake/firedrake/tsfc_interface.py", line 193, in compile_form number_map).kernels File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", line 200, in __new__ obj = make_obj() File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", line 190, in make_obj obj.__init__(*args, **kwargs) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ src/firedrake/firedrake/tsfc_interface.py", line 121, in __init__ kernels.append(KernelInfo(kernel=Kernel(ast, ast.name, opts=opts), File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", line 200, in __new__ obj = make_obj() File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", line 190, in make_obj obj.__init__(*args, **kwargs) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/base.py", line 3843, in __init__ self._code = self._ast_to_c(self._ast, opts) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ src/PyOP2/pyop2/sequential.py", line 73, in _ast_to_c ast_handler.plan_cpu(self._opts) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/COFFEE/coffee/plan.py", line 121, in plan_cpu loop_opt.rewrite(rewrite) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ src/COFFEE/coffee/optimizer.py", line 117, in rewrite ew.sharing_graph_rewrite() File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ src/COFFEE/coffee/rewriter.py", line 619, in sharing_graph_rewrite prob.solve(ilp.GLPK(msg=0)) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ lib/python2.7/site-packages/pulp/pulp.py", line 1651, in solve status = solver.actualSolve(self, **kwargs) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ lib/python2.7/site-packages/pulp/solvers.py", line 383, in actualSolve rc = subprocess.call(proc, stdout = pipe, stderr = pipe) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ lib/python2.7/site-packages/subprocess32.py", line 578, in call p = Popen(*popenargs, **kwargs) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ lib/python2.7/site-packages/subprocess32.py", line 825, in __init__ restore_signals, start_new_session) File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/ lib/python2.7/site-packages/subprocess32.py", line 1574, in _execute_child raise child_exception_type(errno_num, err_msg) OSError: [Errno 14] Bad address Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

4 10

installing firedrake on a cluster
by Francis Poulin 31 Mar '17

31 Mar '17

Hello, I started trying to install firedrake on a cluster and I have some rather silly problems. First, it won't install because PYTHONPATH is installed. That's easy to resolve since I can ask it to honour the python path and then it goes forward. Second, it notices that there is a PETSc directory. I am pretty sure that I won't the install script to build petsc and not use what's there. IS there a way to get it to ignore both of these? I tried PYTHONPATH='' but that didn't seem to help for the first part so I don't imagine it will help much for the second. The good news is that the default python is 2.7.8 Any advice would be greatly appreciated to get by the python and petsc path's already set? Cheers, Francis ------------------ Francis Poulin Associate Professor Department of Applied Mathematics University of Waterloo email: fpoulin(a)uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637

2 5

Re: [firedrake] Fwd: [petsc-maint] Weak scaling test: Fieldsplit questions
by David Ham 30 Mar '17

30 Mar '17

Hi Matt, Are you sure? To me it looks like pulp is using subprocess to fork the linear program solver. Nothing to do with parallel. David On Thu, 30 Mar 2017 at 14:30 Matthew Knepley <knepley(a)gmail.com> wrote: > What is going on here? It looks like you are using subprocess. Why would > you do that on a cluster rather than MPI? > > Matt > > ---------- Forwarded message ---------- > From: *Buesing, Henrik* <HBuesing(a)eonerc.rwth-aachen.de> > Date: Thu, Mar 30, 2017 at 8:25 AM > Subject: AW: [petsc-maint] Weak scaling test: Fieldsplit questions > To: Matthew Knepley <knepley(a)gmail.com> > Cc: Barry Smith <bsmith(a)mcs.anl.gov>, Hong <hzhang(a)mcs.anl.gov>, " > petsc-maint(a)mcs.anl.gov" <petsc-maint(a)mcs.anl.gov> > > > In my opinion, there is some kind of race condition in Firedrake when > running on more than one node. Thus, until this is fixed it is very > unlikely for me to get the 64 cores case running. > > > Hmm, we are running Firedrake in parallel with no problems here. What is > the error? > > > > *[Buesing, Henrik] *See [1] for the error message and the attached three > logs (for the 32 core case this was a 2/5 running and 3/5 crashing ). > > > > This is just for running the compiled code. During the compile stage I had > problems, too. What I did is the following: 1) Run Firedrake on 1 node > (this works). Now all the *.so files are in place. 2) Run Firedrake on more > than one node. This crashes more often the more processes I use. > > > > I’m guessing for a race condition, because on 17 cores (1 node + 1 core) > my problem runs fine. On 32 cores it sometimes runs. And on 64 cores it, up > to now, never runs. > > But if you are not having these problems, and if the provided code > reproduces the MatCreateSubMats problem, then you can do tests on your own. > Well, a lot of ifs, but better than nothing. > > > > Thank you! > Henrik > > > > [1] > > > > Traceback (most recent call last): > > File "/work/hb111949/Firedrake/twophase/2pDrake/2pinjection.py", line > 228, in <module> > > solver = NonlinearVariationalSolver(problem,options_prefix="") > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/variational_solver.py", > line 156, in __init__ > > pre_function_callback=pre_f_callback) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/solving_utils.py", > line 260, in __init__ > > form_compiler_parameters=fcp) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/assemble.py", > line 143, in create_assembly_callable > > collect_loops=True) > > File "<decorator-gen-279>", line 2, in _assemble > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/utils.py", > line 62, in wrapper > > return f(*args, **kwargs) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/assemble.py", > line 192, in _assemble > > kernels = tsfc_interface.compile_form(f, "form", > parameters=form_compiler_parameters, inverse=inverse) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/tsfc_interface.py", > line 193, in compile_form > > number_map).kernels > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", > line 200, in __new__ > > obj = make_obj() > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", > line 190, in make_obj > > obj.__init__(*args, **kwargs) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/tsfc_interface.py", > line 121, in __init__ > > kernels.append(KernelInfo(kernel=Kernel(ast, ast.name, opts=opts), > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", > line 200, in __new__ > > obj = make_obj() > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", > line 190, in make_obj > > obj.__init__(*args, **kwargs) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/base.py", > line 3843, in __init__ > > self._code = self._ast_to_c(self._ast, opts) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/sequential.py", > line 73, in _ast_to_c > > ast_handler.plan_cpu(self._opts) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/COFFEE/coffee/plan.py", > line 121, in plan_cpu > > loop_opt.rewrite(rewrite) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/COFFEE/coffee/optimizer.py", > line 117, in rewrite > > ew.sharing_graph_rewrite() > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/COFFEE/coffee/rewriter.py", > line 619, in sharing_graph_rewrite > > prob.solve(ilp.GLPK(msg=0)) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/lib/python2.7/site-packages/pulp/pulp.py", > line 1651, in solve > > status = solver.actualSolve(self, **kwargs) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/lib/python2.7/site-packages/pulp/solvers.py", > line 383, in actualSolve > > rc = subprocess.call(proc, stdout = pipe, stderr = pipe) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/lib/python2.7/site-packages/subprocess32.py", > line 578, in call > > p = Popen(*popenargs, **kwargs) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/lib/python2.7/site-packages/subprocess32.py", > line 825, in __init__ > > restore_signals, start_new_session) > > File > "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/lib/python2.7/site-packages/subprocess32.py", > line 1574, in _execute_child > > raise child_exception_type(errno_num, err_msg) > > OSError: [Errno 14] Bad address > > Thanks, > > > > Matt > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Dr David Ham Department of Mathematics Imperial College London

1 0

Re: [firedrake] close to fixing my nonlinear QG solver
by Colin Cotter 30 Mar '17

30 Mar '17

It's a classic gotcha. We are kind of stuck with it due to wanting to stay close to the Fenics interface, but maybe we should have a specific bit in the docs about it [or a more highlighted one, if it already exists]. all the best --cjc On 30 March 2017 at 12:57, Francis Poulin <fpoulin(a)uwaterloo.ca> wrote: > Houston, we have convergence! > > Thank you for the help Lawrence. > > I did see there were emails about the split passed around last week, which > I clearly should have read more closely. > > Now that I have convergence on a course grid I will test a finder grid and > let you know how robust this is. > > Cheers, Francis > > ------------------ > Francis Poulin > Associate Professor > Department of Applied Mathematics > University of Waterloo > > email: fpoulin(a)uwaterloo.ca > Web: https://uwaterloo.ca/poulin-research-group/ > Telephone: +1 519 888 4567 x32637 > > > ________________________________________ > From: firedrake-bounces(a)imperial.ac.uk [firedrake-bounces(a)imperial.ac.uk] > on behalf of Lawrence Mitchell [lawrence.mitchell(a)imperial.ac.uk] > Sent: Thursday, March 30, 2017 7:14 AM > To: firedrake(a)imperial.ac.uk > Subject: Re: [firedrake] close to fixing my nonlinear QG solver > > Hi Francis, > > sorry, I now saw your problem. > > You do: > > soln1.assign(soln0) > > psi1, u1, q1 = soln1.split() > > Gnon = F(psi1, u1, q1) > > You've been bitten by the "wrong split" bug!! ARGH. > > Only use function.split() for output. If you're going to use the > split values in a form, you need to use "split(function)" > > If you replace your lines: > > psi1, u1, q1 = soln1.split() > Gnon = ... > > with: > > psi1, u1, q1 = split(soln1) > Gnon = ... > psi1, u1, q1 = soln1.split() > > > Everything works. > > Lawrence > > > _______________________________________________ > firedrake mailing list > firedrake(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/firedrake > -- http://www.imperial.ac.uk/people/colin.cotter www.cambridge.org/9781107663916

2 1

close to fixing my nonlinear QG solver
by Francis Poulin 30 Mar '17

30 Mar '17

Hello Firedrakers, Thanks everyone for a very interesting Workshop. It was great to meet everyone that was able to attend and learn about all the diverse range of problems people are working on. I have a question to follow up on discussions from the workshop. I am not sure if this quite of question is appropriate for Slack so I thought I would send an email. If you prefer Slack I will happily go there next time. On Wednesday morning David pointed out that my nonlinear solver for the Quasi-Geostrophic model has a problem in that I computed the Laplacian of a field, which is one too many derivatives. I now know that's bad and think this could really fix up my convergence issues. Thanks David! I rewrote the problem in terms of a velocity, a vorticity and a streamfunction and coded it up. There are three fields instead of one so it is bigger but at least I don't get any double derivatives. I am pretty sure that the linear part is correct since the solution looks identical to what I had before. Unfortunately, after spending a lot of time on the nonlinear problem, I can't seem to get the syntax right. Even when I try a much simpler nonlinear term it still gives me the same error. The version that I have now can be found at the following link: https://github.com/francispoulin/firedrakeQG/blob/master/QG1L_Stommel_psiuq… The error can be found below. I tried putting a Constant(0)* in front of the nonlinear term, like I think Lawrence showed me how to do, but I seemed to get the syntax wrong on that as well. The fact that Linear and Nonlinear firedrake fail to converge means that it's not even trying because I messed up something. Any advice on how I can fix up my syntax? Cheers, Francis $ python QG1L_Stommel_psiuq.py Linear firedrake_1_ solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0 PCSETUP_FAILED due to FACTOR_NUMERIC_ZEROPIVOT Nonlinear firedrake_1_ solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 Traceback (most recent call last): File "QG1L_Stommel_psiuq.py", line 114, in <module> nonlinear_solver.solve() File "/home/fpoulin/software/firedrake/src/firedrake/firedrake/variational_solver.py", line 223, in solve solving_utils.check_snes_convergence(self.snes) File "/home/fpoulin/software/firedrake/src/firedrake/firedrake/solving_utils.py", line 160, in check_snes_convergence %s""" % (snes.getIterationNumber(), msg)) firedrake.exceptions.ConvergenceError: Nonlinear solve failed to converge after 0 nonlinear iterations. Reason: Inner linear solve failed to converge after 0 iterations with reason: unknown reason (petsc4py enum incomplete?), try with -snes_convered_reason and -ksp_converged_reason ------------------ Francis Poulin Associate Professor Department of Applied Mathematics University of Waterloo email: fpoulin(a)uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637

2 4