- firedrake - imperial.ac.uk

Error on Installation
by Amireh Mousavi 30 May '18

30 May '18

Dear all, I tried to install Firedrake on Ubuntu 18.04 but I have problem on installation. Here are the errors I get: Running firedrake-install Checking for presence of package build-essential... installed. Checking for presence of package autoconf... installed. Checking for presence of package automake... installed. Checking for presence of package cmake... installed. Checking for presence of package gfortran... installed. Checking for presence of package git-core... missing. Checking for presence of package libblas-dev... installed. Checking for presence of package liblapack-dev... installed. Checking for presence of package libopenmpi-dev... installed. Checking for presence of package libtool... installed. Checking for presence of package mercurial... installed. Checking for presence of package openmpi-bin... installed. Checking for presence of package python3-dev... installed. Checking for presence of package python3-pip... installed. Checking for presence of package python3-tk... installed. Checking for presence of package python3-venv... installed. Checking for presence of package zlib1g-dev... installed. Installing missing packages: git-core. [sudo] password for amireh: Reading package lists... Done Building dependency tree Reading state information... Done Note, selecting 'git' instead of 'git-core' git is already the newest version (1:2.17.0-1ubuntu1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Creating firedrake venv in '/home/amireh/firedrake'. Cloning firedrake Failed to clone firedrake using ssh, falling back to https. Successfully cloned repository firedrake. Checking out branch master Successfully checked out branch master Cloning the dependencies of firedrake Cloning ufl Failed to clone ufl using ssh, falling back to https. Successfully cloned repository ufl. Checking out branch master Successfully checked out branch master Cloning fiat Failed to clone fiat using ssh, falling back to https. Successfully cloned repository fiat. Checking out branch master Successfully checked out branch master Cloning FInAT Failed to clone FInAT using ssh, falling back to https. Successfully cloned repository FInAT. Checking out branch master Successfully checked out branch master Cloning tsfc Failed to clone tsfc using ssh, falling back to https. Successfully cloned repository tsfc. Checking out branch master Successfully checked out branch master Cloning PyOP2 Failed to clone PyOP2 using ssh, falling back to https. Successfully cloned repository PyOP2. Checking out branch master Successfully checked out branch master Cloning the dependencies of PyOP2 Cloning petsc Failed to clone petsc using ssh, falling back to https. Successfully cloned repository petsc. Checking out branch firedrake Successfully checked out branch firedrake Cloning petsc4py Failed to clone petsc4py using ssh, falling back to https. Successfully cloned repository petsc4py. Checking out branch firedrake Successfully checked out branch firedrake Cloning COFFEE Failed to clone COFFEE using ssh, falling back to https. Successfully cloned repository COFFEE. Checking out branch master Successfully checked out branch master Installing petsc/ Downloading Eigen from 'https://bitbucket.org/eigen/eigen/get/3.3.3.tar.gz' to 'eigen-3.3.3.tgz' Checking Eigen tarball integrity Eigen tarball hash valid Building PETSc. Depending on your platform, may take between a few minutes and an hour or more to build! Traceback (most recent call last): File "firedrake-install", line 1074, in <module> install("petsc/") File "firedrake-install", line 554, in install build_and_install_petsc() File "firedrake-install", line 660, in build_and_install_petsc run_pip_install(["--ignore-installed", "petsc/"]) File "firedrake-install", line 529, in run_pip_install check_call(pipinstall + pipargs) File "firedrake-install", line 366, in check_call log.debug(subprocess.check_output(arguments, stderr=subprocess.STDOUT, env=env).decode()) File "/usr/lib/python3.6/subprocess.py", line 336, in check_output **kwargs).stdout File "/usr/lib/python3.6/subprocess.py", line 418, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['/home/amireh/firedrake/bin/python', '-m', 'pip', 'install', '--no-binary', 'mpi4py', '--no-deps', '-vvv', '--ignore-installed', 'petsc/']' returned non-zero exit status 1. Install log saved in firedrake-install.log Please guide me to install. Best regards, Amireh

2 4

Error on Installation
by Amireh Mousavi 30 May '18

30 May '18

Dear all, I tried to install Firedrake on Ubuntu 18.04 but I have problem on installation. Here are the errors we get: Running firedrake-install Checking for presence of package build-essential... installed. Checking for presence of package autoconf... installed. Checking for presence of package automake... installed. Checking for presence of package cmake... installed. Checking for presence of package gfortran... installed. Checking for presence of package git-core... missing. Checking for presence of package libblas-dev... installed. Checking for presence of package liblapack-dev... installed. Checking for presence of package libopenmpi-dev... installed. Checking for presence of package libtool... installed. Checking for presence of package mercurial... installed. Checking for presence of package openmpi-bin... installed. Checking for presence of package python3-dev... installed. Checking for presence of package python3-pip... installed. Checking for presence of package python3-tk... installed. Checking for presence of package python3-venv... installed. Checking for presence of package zlib1g-dev... installed. Installing missing packages: git-core. [sudo] password for amireh: Reading package lists... Done Building dependency tree Reading state information... Done Note, selecting 'git' instead of 'git-core' git is already the newest version (1:2.17.0-1ubuntu1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Creating firedrake venv in '/home/amireh/firedrake'. Cloning firedrake Failed to clone firedrake using ssh, falling back to https. Successfully cloned repository firedrake. Checking out branch master Successfully checked out branch master Cloning the dependencies of firedrake Cloning ufl Failed to clone ufl using ssh, falling back to https. Successfully cloned repository ufl. Checking out branch master Successfully checked out branch master Cloning fiat Failed to clone fiat using ssh, falling back to https. Successfully cloned repository fiat. Checking out branch master Successfully checked out branch master Cloning FInAT Failed to clone FInAT using ssh, falling back to https. Successfully cloned repository FInAT. Checking out branch master Successfully checked out branch master Cloning tsfc Failed to clone tsfc using ssh, falling back to https. Successfully cloned repository tsfc. Checking out branch master Successfully checked out branch master Cloning PyOP2 Failed to clone PyOP2 using ssh, falling back to https. Successfully cloned repository PyOP2. Checking out branch master Successfully checked out branch master Cloning the dependencies of PyOP2 Cloning petsc Failed to clone petsc using ssh, falling back to https. Successfully cloned repository petsc. Checking out branch firedrake Successfully checked out branch firedrake Cloning petsc4py Failed to clone petsc4py using ssh, falling back to https. Successfully cloned repository petsc4py. Checking out branch firedrake Successfully checked out branch firedrake Cloning COFFEE Failed to clone COFFEE using ssh, falling back to https. Successfully cloned repository COFFEE. Checking out branch master Successfully checked out branch master Installing petsc/ Downloading Eigen from 'https://bitbucket.org/eigen/eigen/get/3.3.3.tar.gz' to 'eigen-3.3.3.tgz' Checking Eigen tarball integrity Eigen tarball hash valid Building PETSc. Depending on your platform, may take between a few minutes and an hour or more to build! Traceback (most recent call last): File "firedrake-install", line 1074, in <module> install("petsc/") File "firedrake-install", line 554, in install build_and_install_petsc() File "firedrake-install", line 660, in build_and_install_petsc run_pip_install(["--ignore-installed", "petsc/"]) File "firedrake-install", line 529, in run_pip_install check_call(pipinstall + pipargs) File "firedrake-install", line 366, in check_call log.debug(subprocess.check_output(arguments, stderr=subprocess.STDOUT, env=env).decode()) File "/usr/lib/python3.6/subprocess.py", line 336, in check_output **kwargs).stdout File "/usr/lib/python3.6/subprocess.py", line 418, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['/home/amireh/firedrake/bin/python', '-m', 'pip', 'install', '--no-binary', 'mpi4py', '--no-deps', '-vvv', '--ignore-installed', 'petsc/']' returned non-zero exit status 1. Install log saved in firedrake-install.log Please guide me to install. Best regards, Amireh

1 0

Poor strong scaling
by Siddhant Wahal 23 May '18

23 May '18

Hello, I'm trying to reproduce the firedrake strong scaling experiments, as reported in this paper <https://arxiv.org/pdf/1501.01809.pdf>. However, I'm unable to reproduce similar scaling trends in my experiments. Here are some more details: *Machine:* I'm on TACC's stampede2, and running experiments on the skylake nodes. Each node has 2 sockets, and 24 physical cores on each socket. *PETSc:* The firedrake fork of PETSc was compiled with the 2017 intel compilers and the -O3 and skylake specific -xCORE-AVX512 flags. *Problem:* The problem is the poisson equation in 3D. The preconditioner and linear solver are again the same as in the paper. At each core count, each problem was run twice and only the second run was timed. *Results:* I'm attaching a strong scaling plot. This plot was created using 80x80x80 cubes in the mesh, and 3rd order continuous polynomials as the finite element function space. This results in ~14M DOFs in total. The plot shows the maximum time spent solving the pre-assembled linear system, as measured by wrapping MPI timers around firedrake.solve(A, u, solver_parameters=param). Also reported is the time spent in the KSPSolve event. While the KSPSolve event scales well, the call to firedrake.solve doesn't. It seems like operation that occurs between firedrake.solve and before ksp.solve is causing the bottleneck. I'm also attaching the python script I used to solve the Poisson equation, and a sample output from PETSc's -log_view option. Any hints as to why this isn't scaling will be appreciated!

2 2

Re: [firedrake] Intel MKL error on skylake
by Siddhant Wahal 23 May '18

23 May '18

Hi Justin, I had the same error, and fixed it by setting the LD_PRELOAD environment variable- export LD_PRELOAD=$mklroot/lib/intel64_lin/libmkl_avx512.so:$mklroot/lib/intel64_lin/libmkl_def.so Here, $mklroot is the the path to the MKL installation on your cluster. > Hi all, > > I'm trying to install firedrake onto a skylake cluster, and I have > to not only install firedrake as a local user but also to install > my own python3. So PETSc, firedrake, and python3 is built using > intel compilers. > > So everything installs correctly, and nothing terrible happens when > i type > > python -c "from firedrake import *" > > however, when I try to run any firedrake code, I get this error: > > Intel MKL FATAL ERROR: Cannot load libmkl_avx512.so or > libmkl_def.so.

1 0

Re: [firedrake] Intel MKL error on skylake
by Lawrence Mitchell 23 May '18

23 May '18

Hi Justin, (you original email didn't come through, because all the message attachments were too large). I attach them here for posterity (gzipped). You wrote: > Hi all, > > I'm trying to install firedrake onto a skylake cluster, and I have > to not only install firedrake as a local user but also to install > my own python3. So PETSc, firedrake, and python3 is built using > intel compilers. > > So everything installs correctly, and nothing terrible happens when > i type > > python -c "from firedrake import *" > > however, when I try to run any firedrake code, I get this error: > > Intel MKL FATAL ERROR: Cannot load libmkl_avx512.so or > libmkl_def.so. I hate MKL, why they cannot just provide a library to link against is beyond me. Anyway, I wonder if this is to do with numpy (which the install script installs) shipping with its own BLAS (rather than building against a user-provided BLAS). Can you do simple numpy things? import numpy as np u, s, v = np.linalg.svd(np.arange(1, 10).reshape(3, 3)) If so, then you can try narrowing down where the exact error occurs. Can you run any petsc4py demos on their own? Can you run pyop2 tests on their own? Where do you get the firedrake errors? Can you make meshes? What about function spaces? etc... Cheers, Lawrence

1 0

errornorm()
by Matthew Knepley 21 May '18

21 May '18

This should do the 1-norm and the \infty-norm Thanks, Matt & Justin -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>

2 5

mesh coordinates and boundaries questions
by Ed Bueler 21 May '18

21 May '18

Dear Firedrake -- I have two questions related to mesh coordinates and boundaries: 1) How does one get the coordinates of the points on a boundary? That is, suppose I know the id number of that part, for example "id" as in "ds(id)" used in forming boundary terms in the weak form. What is the corresponding idiom for getting the spatial coordinates? Something like "mesh.coordinates(id)" (though that does not work)? In parallel I guess this function will only be defined on a subset of the processors? 2) Given a function along a boundary, how does one extend that to the whole mesh, by holding constant in one variable? In the case I care about the domain is of the special form D = {(x,y) | f(x) < y < g(x)} and one has values w(x) along the y=g(x) boundary. How does one build a Function on the whole mesh which is W(x,y) = w(x) (I understand that my domain is special, namely logically rectangular. And I can imagine an answer specific to an extruded mesh ... but I would like to not go there now.) Again I am interested in a solution which works in parallel. Maybe I have to commit to solving a first-order PDE just to do this? Thanks for your help with my questions so far! Ed -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman

3 7

Re: [firedrake] nonlinear diffusion with space-time Robin condition
by Ham, David A 15 May '18

15 May '18

Hi Onno, As Stephan correctly (as usual) points out, your primary speed issue is that you are using solve instead of LinearVariationalProblem and LinearVariationalSolver. This means you unnecessarily duplicate a whole load of setup every timestep. In this case, the simplest thing to do is to delete your special case for theta = 0 and just have the theta = 0 case run through the nonlinear code path. On my machine this gets you two time units in about two seconds. If you really want to, you can set the solver options in the theta = 0 case to switch off the outer nonlinear solve. However my observation is that this makes negligible additional difference because PETSc only does one solver iteration anyway (it does an extra residual evaluation, but this seems not to be a significant cost). One would not expect Firedrake to be significantly faster than Matlab in this case because the problem is so tiny. The reason that you are observing failure in the theta > 0 case is this term: sqrt(max_value(2.0*h/3.0,0.0)) When this gets differentiated, you get a 1/sqrt(h) term, which has a pole at h=0 (which is not a huge surprise since the expression is actually not differentiable at h=0). This inserts NaNs into the matrix and the solver dies. Firedrake is doing the right thing, it’s just that your problem is not well defined there. You have two options: 1. Use an initial guess which is not zero, and hope that the newton solve never actually hits h=0 at the boundary (which it probably won’t). 2. Change the form to something differentiable (or at least with finite formal derivative) at h=0. Regards, David PS. In many circumstances, solver options are also a significant speed factor. However in this case (a) the tiny problem largely makes this irrelevant and (b) because you have a 1D problem, the default ilu(0) preconditioner acts like a direct solver, which is almost certainly the best option for tiny problems like this. From: <firedrake-bounces(a)imperial.ac.uk> on behalf of Onno Bokhove <O.Bokhove(a)leeds.ac.uk> Reply-To: firedrake <firedrake(a)imperial.ac.uk> Date: Sunday, 13 May 2018 at 13:15 To: firedrake <firedrake(a)imperial.ac.uk> Subject: [firedrake] nonlinear diffusion with space-time Robin condition Dear Firedrakers, (i) Will & I tried to build a theta-time-step solver including Crank-Nicolson for theta=1/2 but only get it to work for theta=0 (Forward Euler); see the attached code at "HERE2" which matches (32) in the attached pdf. [This is really a test set-up atm for a larger question on dynamic boundary conditions for an elliptic problem, but the system may be amusing by itself as a tentative FD-example.) (ii) The working explicit Forward Euler code for theta=0 is very slow, at least 10x slower than the same algorithm in a FEM matlab code (-). See "HERE1" in the attached code; why is the code so slow? [I have also attached the basic matlab code.] This example may be amusing because of the space-time Robin condition (14) applied to (11), leading to the Firedrake weak form (32) --when time is discretised as well. The question can also be solved, slightly differently wrt the implementation of the Robin boundary condition, using finite differences, which makes for an amusing comparison. I am not even sure whether this space-time bc and nonlinear diffusion eqn is well-posed but the finite difference and FEM numerics seems to be consistent. Thank you and best wishes, Onno

3 3

RA position
by Colin Cotter 15 May '18

15 May '18

Dear all, Just to let you know about a postdoc vacancy working with Firedrake on time-parallel methods: https://www.imperial.ac.uk/jobs/description/NAT00175/research-associate-mat… all queries to me offline please. all the best --cjc -- Colin Cotter Department of Mathematics Imperial College London I don’t expect a reply to this email from you outside of your normal working hours. http://www.imperial.ac.uk/people/colin.cotter www.cambridge.org/9781107663916

1 0

did not get your answer because I get firedrake digest only? (re: mesh coordinates and boundaries questions)
by Ed Bueler 14 May '18

14 May '18

David -- *After* I wrote my latest to the firedrake mailing list I found your answer at the archive: http://mailman.ic.ac.uk/pipermail/firedrake/2018-May/003406.html I did not get it before. (Its not in my spam or trash.) Probably because I get only the digest version of the mailing list? Or is there a need to CC me in addition to sending to the list? Not sure what are best practices to avoid crossing this way. Sorry! Regarding the content of your answer: 1) I still don't know how to use the subdomain id to get anything other than the *measure* for the boundary, e.g. "ds(id)". How do I get an indicator function, for example? This is what I am missing as "idiom". 2) How does interpolate work if I have data only locally on the boundary, and not as an analytical function of x, and the data only lives on the process that holds its part of the boundary? That is, I have "values of w(x)" on the boundary (as stated) but not a formula in x (i.e. the "expression" needed for interpolate). Said a different way, the last sentence at https://www.firedrakeproject.org/interpolation.html#id2 namely "This will also work in parallel, as the interpolation will occur on each process,...", can't possibly be true in the situation where values w(x), for x in some range, are only known to the process which holds that part of the boundary (i.e. for x in that range). I think all-to-all communication is needed to answer my question in general. Said yet another way, in the general setting I think that constructing the "mydata(X)" described in same the firedrake doc would require setting up an entire VecScatter or solve() anyway. I think both of your answers assume I have a formula in terms of x. I don't in the application which drove my questions. Thanks, and sorry for the fumble, Ed -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman

2 1