March 2017 - firedrake - imperial.ac.uk

Firedrake in parallel
by Buesing, Henrik 30 Mar '17

30 Mar '17

Dear all, I am having problems running Firdrake in parallel on more than one node. If I run on 16 cores (2x8 cores, 1 node) everything is fine. If I go to more than one node (2,4,...) I get an error (see also attached log): OSError: /w0/tmp/lsf_user.35589703.0/pyop2-cache-uid17470/f51f6074c31bf0cd78d85bcb40381923.so: cannot open shared object file: No such file or directory Do you have an idea what could be wrong? Thank you! Henrik -- Dipl.-Math. Henrik Büsing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing(a)eonerc.rwth-aachen.de ------------------------------------------------------

4 6

Firedrake install error
by William Booker 26 Mar '17

26 Mar '17

Dear Firedrakers, My laptop seems to have issues when the install script attempts to install eigen. What should I do to fix this? Thanks Will Error: =============================================================================== Trying to download hg://https://bitbucket.org/eigen/eigen/ for EIGEN =============================================================================== =============================================================================== Configuring EIGEN with cmake, this may take several minutes =============================================================================== ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error configuring EIGEN with cmake Could not execute "cd /tmp/pip-RTQ3Ml-build/arch-python-linux-x86_64/externalpackages/hg.eigen/build && /usr/bin/cmake .. -DCMAKE_INSTALL_PREFIX=/usr/not-backed-up/firedrake/march/firedrake/lib/python2.7/site-packages/petsc -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_C_COMPILER="/usr/lib64/openmpi/bin/mpicc" -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC -fstack-protector -g -O" -DCMAKE_CXX_COMPILER="/usr/lib64/openmpi/bin/mpicxx" -DCMAKE_CXX_FLAGS:STRING="-fstack-protector -g -O -fPIC" -DCMAKE_Fortran_COMPILER="/usr/lib64/openmpi/bin/mpif90" -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0 -g -O" -DBUILD_SHARED_LIBS=on -DENABLE_OPENMP=OFF": -- The C compiler identification is GNU 4.4.7 -- The CXX compiler identification is GNU 4.4.7 -- Check for working C compiler: /usr/lib64/openmpi/bin/mpicc -- Check for working C compiler: /usr/lib64/openmpi/bin/mpicc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /usr/lib64/openmpi/bin/mpicxx -- Check for working CXX compiler: /usr/lib64/openmpi/bin/mpicxx -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Performing Test standard_math_library_linked_to_automatically -- Performing Test standard_math_library_linked_to_automatically - Success -- Standard libraries to link to explicitly: none -- Performing Test COMPILER_SUPPORT_WERROR -- Performing Test COMPILER_SUPPORT_WERROR - Success -- Performing Test COMPILER_SUPPORT_pedantic -- Performing Test COMPILER_SUPPORT_pedantic - Success -- Performing Test COMPILER_SUPPORT_Wall -- Performing Test COMPILER_SUPPORT_Wall - Success -- Performing Test COMPILER_SUPPORT_Wextra -- Performing Test COMPILER_SUPPORT_Wextra - Success -- Performing Test COMPILER_SUPPORT_Wundef -- Performing Test COMPILER_SUPPORT_Wundef - Success -- Performing Test COMPILER_SUPPORT_Wcastalign -- Performing Test COMPILER_SUPPORT_Wcastalign - Success -- Performing Test COMPILER_SUPPORT_Wcharsubscripts -- Performing Test COMPILER_SUPPORT_Wcharsubscripts - Success -- Performing Test COMPILER_SUPPORT_Wnonvirtualdtor -- Performing Test COMPILER_SUPPORT_Wnonvirtualdtor - Success -- Performing Test COMPILER_SUPPORT_Wunusedlocaltypedefs -- Performing Test COMPILER_SUPPORT_Wunusedlocaltypedefs - Failed -- Performing Test COMPILER_SUPPORT_Wpointerarith -- Performing Test COMPILER_SUPPORT_Wpointerarith - Success -- Performing Test COMPILER_SUPPORT_Wwritestrings -- Performing Test COMPILER_SUPPORT_Wwritestrings - Success -- Performing Test COMPILER_SUPPORT_Wformatsecurity -- Performing Test COMPILER_SUPPORT_Wformatsecurity - Success -- Performing Test COMPILER_SUPPORT_Wshorten64to32 -- Performing Test COMPILER_SUPPORT_Wshorten64to32 - Failed -- Performing Test COMPILER_SUPPORT_Wlogicalop -- Performing Test COMPILER_SUPPORT_Wlogicalop - Success -- Performing Test COMPILER_SUPPORT_Wenumconversion -- Performing Test COMPILER_SUPPORT_Wenumconversion - Failed -- Performing Test COMPILER_SUPPORT_Wcpp11extensions -- Performing Test COMPILER_SUPPORT_Wcpp11extensions - Failed -- Performing Test COMPILER_SUPPORT_Wdoublepromotion -- Performing Test COMPILER_SUPPORT_Wdoublepromotion - Failed -- Performing Test COMPILER_SUPPORT_Wnopsabi -- Performing Test COMPILER_SUPPORT_Wnopsabi - Success -- Performing Test COMPILER_SUPPORT_Wnovariadicmacros -- Performing Test COMPILER_SUPPORT_Wnovariadicmacros - Success -- Performing Test COMPILER_SUPPORT_Wnolonglong -- Performing Test COMPILER_SUPPORT_Wnolonglong - Success -- Performing Test COMPILER_SUPPORT_fnochecknew -- Performing Test COMPILER_SUPPORT_fnochecknew - Success -- Performing Test COMPILER_SUPPORT_fnocommon -- Performing Test COMPILER_SUPPORT_fnocommon - Success -- Performing Test COMPILER_SUPPORT_fstrictaliasing -- Performing Test COMPILER_SUPPORT_fstrictaliasing - Success -- Performing Test COMPILER_SUPPORT_wd981 -- Performing Test COMPILER_SUPPORT_wd981 - Failed -- Performing Test COMPILER_SUPPORT_wd2304 -- Performing Test COMPILER_SUPPORT_wd2304 - Failed -- Performing Test COMPILER_SUPPORT_STRICTANSI -- Performing Test COMPILER_SUPPORT_STRICTANSI - Failed -- Performing Test COMPILER_SUPPORT_Qunusedarguments -- Performing Test COMPILER_SUPPORT_Qunusedarguments - Failed -- Performing Test COMPILER_SUPPORT_ansi -- Performing Test COMPILER_SUPPORT_ansi - Success -- Performing Test COMPILER_SUPPORT_OPENMP -- Performing Test COMPILER_SUPPORT_OPENMP - Success -- Performing Test EIGEN_COMPILER_SUPPORT_CPP11 -- Performing Test EIGEN_COMPILER_SUPPORT_CPP11 - Failed -- Looking for Q_WS_X11 -- Looking for Q_WS_X11 - found -- Looking for Q_WS_WIN -- Looking for Q_WS_WIN - not found -- Looking for Q_WS_QWS -- Looking for Q_WS_QWS - not found -- Looking for Q_WS_MAC -- Looking for Q_WS_MAC - not found -- Found Qt4: /usr/bin/qmake-qt4 (found version "4.6.2") -- Performing Test EIGEN_COMPILER_SUPPORT_CXX11 -- Performing Test EIGEN_COMPILER_SUPPORT_CXX11 - Failed -- The Fortran compiler identification is GNU -- Check for working Fortran compiler: /usr/lib64/openmpi/bin/mpif90 -- Check for working Fortran compiler: /usr/lib64/openmpi/bin/mpif90 -- works -- Detecting Fortran compiler ABI info -- Detecting Fortran compiler ABI info - done -- Checking whether /usr/lib64/openmpi/bin/mpif90 supports Fortran 90 -- Checking whether /usr/lib64/openmpi/bin/mpif90 supports Fortran 90 -- yes -- Could NOT find CHOLMOD (missing: CHOLMOD_INCLUDES CHOLMOD_LIBRARIES) -- Could NOT find UMFPACK (missing: UMFPACK_INCLUDES UMFPACK_LIBRARIES) -- Could NOT find SUPERLU (missing: SUPERLU_INCLUDES SUPERLU_LIBRARIES SUPERLU_VERSION_OK) -- Could NOT find PASTIX (missing: PASTIX_INCLUDES PASTIX_LIBRARIES) -- Could NOT find SCOTCH (missing: SCOTCH_INCLUDES SCOTCH_LIBRARIES) -- Could NOT find METIS (missing: METIS_INCLUDES METIS_LIBRARIES) -- Could NOT find SPQR (missing: SPQR_INCLUDES SPQR_LIBRARIES) -- Performing Test COMPILER_SUPPORT_FASTMATH -- Performing Test COMPILER_SUPPORT_FASTMATH - Success -- Configuring incomplete, errors occurred! See also "/tmp/pip-RTQ3Ml-build/arch-python-linux-x86_64/externalpackages/hg.eigen/build/CMakeFiles/CMakeOutput.log". See also "/tmp/pip-RTQ3Ml-build/arch-python-linux-x86_64/externalpackages/hg.eigen/build/CMakeFiles/CMakeError.log".CMake Error at /usr/lib64/boost/Boost.cmake:536 (message): The imported target "boost_date_time-static" references the file "/usr/lib64/lib64/libboost_date_time.a" but this file does not exist. Possible reasons include: * The file was deleted, renamed, or moved to another location. * An install or uninstall procedure did not complete successfully. * The installation package was faulty and contained "/usr/lib64/boost/Boost.cmake" but not all the files it references. Call Stack (most recent call first): /usr/lib64/boost/BoostConfig.cmake:28 (include) /usr/share/cmake/Modules/FindBoost.cmake:177 (find_package) test/CMakeLists.txt:336 (find_package) ******************************************************************************* Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-RTQ3Ml-build/setup.py", line 302, in <module> **metadata) File "/usr/local/modules/python/2.7.9/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/local/modules/python/2.7.9/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/local/modules/python/2.7.9/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/tmp/pip-RTQ3Ml-build/setup.py", line 218, in run config(prefix, self.dry_run) File "/tmp/pip-RTQ3Ml-build/setup.py", line 148, in config if status != 0: raise RuntimeError(status) RuntimeError: 256 ---------------------------------------- Command "/usr/not-backed-up/firedrake/march/firedrake/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-RTQ3Ml-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-teBSaw-record/install-record.txt --single-version-externally-managed --compile --install-headers /usr/not-backed-up/firedrake/march/firedrake/include/site/python2.7/petsc" failed with error code 1 in /tmp/pip-RTQ3Ml-build/ Traceback (most recent call last): File "firedrake-install", line 1231, in <module> install("petsc/") File "firedrake-install", line 675, in install run_pip_install(["--ignore-installed", package]) File "firedrake-install", line 545, in run_pip_install check_call(pipinstall + pipargs) File "firedrake-install", line 366, in check_call subprocess.check_call(arguments, env=env) File "/usr/local/modules/python/2.7.9/lib/python2.7/subprocess.py", line 540, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/usr/not-backed-up/firedrake/march/firedrake/bin/pip', 'install', '--no-binary', 'mpi4py', '--no-deps', '--ignore-installed', 'petsc/']' returned non-zero exit status 1

1 0

Re: [firedrake] Nonlinear solve did not converge due to DIVERGED_FNORM_NAN
by Andrew McRae 24 Mar '17

24 Mar '17

Initial thought: could you be generating a NaN in your own code as the mesh is refined? E.g. if you're interpolating some expression that leads to a 0/0 at some points in the domain. On 24 March 2017 at 10:45, Buesing, Henrik <HBuesing(a)eonerc.rwth-aachen.de> wrote: > Dear all, > > > > I am doing a refinement study for two-phase flow in porous media. The > first five levels of refinement succeed and I get reasonable results. On > level six I get an error (see [1]). I also attached the trace. > > I tried the suggestion with –snes_converged_reason and > –ksp_converged_reason, but this gives no output. Since the error seems to > originate from the projection on the linear output space, I removed these > parts. Then I get [2]. > > Do you have an idea what I am doing wrong? Why does the formulation seem > to be correct for coarse discretizations, but gives NAN for finer ones? > > Thank you! > > Henrik > > > > > > [1] > > firedrake.exceptions.ConvergenceError: Nonlinear solve failed to converge > after 0 nonlinear iterations. > > Reason: > > Inner linear solve failed to converge after 0 iterations with reason: > unknown reason (petsc4py enum incomplete?), try with -snes_convered_reason > and -ksp_converged_reason > > %s""" % (snes.getIterationNumber(), msg)) > > > > [2] > > Nonlinear solve did not converge due to DIVERGED_FNORM_NAN iterations 0 > > > > -- > > Dipl.-Math. Henrik Büsing > > Institute for Applied Geophysics and Geothermal Energy > > E.ON Energy Research Center > > RWTH Aachen University > > ------------------------------------------------------ > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > <+49%20241%208049907> > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > <+49%20241%208049889> > > ------------------------------------------------------ > > http://www.eonerc.rwth-aachen.de/GGE > > hbuesing(a)eonerc.rwth-aachen.de > > ------------------------------------------------------ > > >

3 4

imposing BCs on msh grid
by Francis Poulin 24 Mar '17

24 Mar '17

Hello, When I define a rectanglemesh, as say the following, mesh = RectangleMesh(n0, n0, Lx, Ly, reorder = None) I know that I can impose zero Dirichlet boundary conditions using bc = DirichletBC(Vcg, 0.0, "on_boundary") However, if I impose a mesh created by gmsh (the circle I asked about last week) say using mesh = Mesh("circle.msh") I get the right geometry but it doesn't impose the zero boundary conditions. Is there something you need to do to impose these boundary conditions? Cheers, Francis ------------------ Francis Poulin Associate Professor Department of Applied Mathematics University of Waterloo email: fpoulin(a)uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637

2 2

Re: [firedrake] Nonlinear solve did not converge due to DIVERGED_FNORM_NAN
by Andrew McRae 24 Mar '17

24 Mar '17

"jump" gives the 'jump' in a discontinuous function across a facet of the mesh (to be used inside an interior facet integral). But since you interpolate "x[0]", "x[1]" and "x[2]" (which are obviously continuous) into the discontinuous space, I would expect all the jump terms to evaluate to 0, so Delta_h_facet is 0. If you're dividing by that, this probably explains your problem! I suspect you were trying to do something different... so, what is it that you're trying to do? On 24 March 2017 at 11:52, Buesing, Henrik <HBuesing(a)eonerc.rwth-aachen.de> wrote: > Initial thought: could you be generating a NaN in your own code as the > mesh is refined? E.g. if you're interpolating some expression that leads > to a 0/0 at some points in the domain. > > > > [Buesing, Henrik] I am dividing by Delta_h_facet (see [1]). So for the > cell in the origin, the node should move closer to zero. But, since I am > using DQ0 it should not be zero, right? > > > I am also dividing by dt. Should I reformulate s.t. I multiply by dt? > > Thank you! > > Henrik > > > > > > [1] > > x_func_expr = Expression("x[0]") > > y_func_expr = Expression("x[1]") > > z_func_expr = Expression("x[2]") > > > > x_func = interpolate(x_func_expr, DG1) > > y_func = interpolate(y_func_expr, DG1) > > z_func = interpolate(z_func_expr, DG1) > > Delta_h_facet = sqrt(jump(x_func)**2 > > + jump(y_func)**2 > > + jump(z_func)**2) >

3 5

Firedrake workshop deadline 10 February
by David Ham 24 Mar '17

24 Mar '17

Dear all, This is a gentle reminder that the abstract and early registration deadline for the Firedrake users and developers workshop is next week already! As a reminder, the workshop will be at Imperial on 27 and 28 March, and we encourage users to present talks or posters showing how you are exploiting Firedrake (and possibly highlighting what you'd like us to add). We'll also be incorporating a Firedrake tutorial aimed at new Firedrake users on the Monday morning. New users who wish to only attend the tutorial can do so for free, but they still need to register so we can plan for the right number of participants. Full details on the workshop, registration, and abstract submission are available at: http://www.firedrakeproject.org/firedrake_17.html I hope to see many of you there. Regards, David -- Dr David Ham Department of Mathematics Imperial College London

3 2

Re: [firedrake] Reduce number of output files for parallel runs
by Andrew McRae 24 Mar '17

24 Mar '17

This is expected behaviour in Firedrake. Parallel vtu output consists of a collection of files, each containing data for a different part of the domain. It seems the most natural thing is for each process to write out data for the part of the domain it owns, otherwise we have to do extra moving of data. If we took this to the extreme of moving all the data onto one process, this would create a horrible serial bottleneck (and would be impossible if there was more data than would fit on a node). Why is the large number of files a problem? On 24 March 2017 at 08:52, Buesing, Henrik <HBuesing(a)eonerc.rwth-aachen.de> wrote: > Dear all, > > > > Is it possible to reduce the number of output files in Firedrake when > running in parallel? I am seeing an increase in the number of output files > when increasing the number of processes used. Is this normal behavior? > > Thank you! > > Henrik > > > > > > -- > > Dipl.-Math. Henrik Büsing > > Institute for Applied Geophysics and Geothermal Energy > > E.ON Energy Research Center > > RWTH Aachen University > > ------------------------------------------------------ > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > <+49%20241%208049907> > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > <+49%20241%208049889> > > ------------------------------------------------------ > > http://www.eonerc.rwth-aachen.de/GGE > > hbuesing(a)eonerc.rwth-aachen.de > > ------------------------------------------------------ > > >

2 1

Nonlinear solve did not converge due to DIVERGED_FNORM_NAN
by Buesing, Henrik 24 Mar '17

24 Mar '17

Dear all, I am doing a refinement study for two-phase flow in porous media. The first five levels of refinement succeed and I get reasonable results. On level six I get an error (see [1]). I also attached the trace. I tried the suggestion with -snes_converged_reason and -ksp_converged_reason, but this gives no output. Since the error seems to originate from the projection on the linear output space, I removed these parts. Then I get [2]. Do you have an idea what I am doing wrong? Why does the formulation seem to be correct for coarse discretizations, but gives NAN for finer ones? Thank you! Henrik [1] firedrake.exceptions.ConvergenceError: Nonlinear solve failed to converge after 0 nonlinear iterations. Reason: Inner linear solve failed to converge after 0 iterations with reason: unknown reason (petsc4py enum incomplete?), try with -snes_convered_reason and -ksp_converged_reason %s""" % (snes.getIterationNumber(), msg)) [2] Nonlinear solve did not converge due to DIVERGED_FNORM_NAN iterations 0 -- Dipl.-Math. Henrik Büsing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing(a)eonerc.rwth-aachen.de ------------------------------------------------------

1 0

Reduce number of output files for parallel runs
by Buesing, Henrik 24 Mar '17

24 Mar '17

Dear all, Is it possible to reduce the number of output files in Firedrake when running in parallel? I am seeing an increase in the number of output files when increasing the number of processes used. Is this normal behavior? Thank you! Henrik -- Dipl.-Math. Henrik Büsing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing(a)eonerc.rwth-aachen.de ------------------------------------------------------

1 0

Firedrake PETSc options_left
by Buesing, Henrik 23 Mar '17

23 Mar '17

Dear all, PETSc options "-options_left" and "-log_view" seem to have no effect when setting as solver_parameters. I can prescribe them from the command line, but then I do not get the Firedrake/PETSc "options_left". How can I print the Firedrake/PETSc options_left. Thank you! Henrik -- Dipl.-Math. Henrik Büsing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing(a)eonerc.rwth-aachen.de ------------------------------------------------------

2 3