Re: [firedrake] Installing firedrake on a HPC system without package manager

30 Aug 2019

      Hello again,
Its Friday afternoon, so I'm back poking at the Firedrake/HPC/no apt problem.

First, thanks for all the helpful replies.
...
building Python from source is far easier than
manually installing Firedrake.
True! I built it without any problems using gcc, and with only minor issues using icc. So, I am now Conda free - thanks!
However, even with it firedrake-install fails when it tries to build Petsc. Specifically, it couldn't find the BLAS, and so needs Petsc to be installed with  --download-fblaslapack=1
But I can't see how to pass that option to the firedrake installer.

So, I'm back to installing Petsc by hand, which **seems** to work OK.
No error is reported in by configure; make; make install. I can run all the demos in
/SOMWHERE/petsc/arch-linux-c-debug/externalpackages/git.petsc4py/demo
So that's good (right?).

Next...
...
I think this is actually easy to override: we only need to interpose in one place.
Can you try using the firedrake-install script from here: https://raw.githubusercontent.com/firedrakeproject/firedrake/wence/fix/insta...
You can then do:
export HDF5_DIR=/path/you/used/for/--with-hdf5-dir
python3 firedrake-install --honour-petsc-dir
Thanks for this. It certainly helps. Although installation still fails, at least the log file (attached) is twice as long. Up to 48k lines from my previous best of 21k lines. Progress!
It seems to break when installing PyOP.
Specifically,
 /MYOB/with-my-python/firedrake/bin/python -m pip install --no-binary mpi4py,randomgen -vvv -r PyOP2/requirements-ext.txt
The issue here seems to be with mpi4py.
Frustratingly,
    python -m pip install --no-binary mpi4py,randomgen -vvv -r PyOP2/requirements-ext.txt
succeeds.
Thoughts?

FYI,
python3 firedrake-install -v --disable-ssh --no-package-manager --mpicc mpiicc --mpicxx mpiicpc --mpif90 mpiifort --mpiexec mpiexec --honour-pythonpath --honour-petsc-dir --show-petsc-configure-options
returns with
*********************************************
Would build PETSc with the following options:
*********************************************
--download-chaco
--download-eigen=/ichec/home/users/niall/with-my-python/firedrake/src/eigen-3.3.3.tgz
--download-exodusii
--download-hdf5
--download-hypre
--download-metis
--download-ml
--download-mumps
--download-netcdf
--download-pastix
--download-pnetcdf
--download-pragmatic
--download-ptscotch
--download-scalapack
--download-suitesparse
--download-superlu_dist
--with-cxx-dialect=C++11
--with-fortran-bindings=0
--with-zlib
Eigen will be downloaded from https://bitbucket.org/eigen/eigen/get/3.3.3.tar.gz
---end---
...
apologies that this is still not working, although we would like to get to the bottom.
No apologies needed! I can still do dev work on my Ubuntu laptop, where firedrake installed out-of-the-box. Getting it running in the cluster will be great, eventually, but not urgent.
Have a good weekend.
Niall.

Dr Niall Madden,
School of Mathematics, Statistics and Applied Mathematics, NUI Galway
Vice-dean for Equality, Diversity and Inclusion, College of Science
Web: http://www.maths.nuigalway.ie/~niall
Local co-organiser: 23rd Conference of the International Linear Algebra Society - www.ilas2020.ie<http://ilas2020.ie/>
Associate Editor of Numerical Algorithms (Springer)
________________________________
From: Ham, David A <david.ham@imperial.ac.uk>
Sent: Friday 23 August 2019 11:41
To: Madden, Niall <niall.madden@nuigalway.ie>; Lawrence Mitchell <wence@gmx.li>
Cc: Sagiyama, Koki <k.sagiyama@imperial.ac.uk>; firedrake <firedrake@imperial.ac.uk>
Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager

When you compiled PETSc by hand, which configure options did you use? We depend on PETSc having downloaded quite a few things for us, in particular including hdf5 . It looks like you didn’t build PETSc with that option.

If you run:

firedrake-install --show-petsc-options

the installer will list the options it would have passed to PETSc’s configure. You should ensure that you at least pass those options.

Regards,

David

From: "Madden, Niall" <niall.madden@nuigalway.ie>
Date: Thursday, 22 August 2019 at 22:57
To: "Ham, David A" <david.ham@imperial.ac.uk>, Lawrence Mitchell <wence@gmx.li>
Cc: "Sagiyama, Koki" <k.sagiyama@imperial.ac.uk>, firedrake <firedrake@imperial.ac.uk>
Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager

Hi guys,

Thanks for the replies.

Lawrence:
...
Given you started with a clean conda environment, did you run into
   issues just running firedrake-install inside that?
Yes. Fails when compiling Petsc.

Since I can compile Petsc by hand, I tried that passed --honour-petsc-dir when trying to build firedrake.

But then it fails to compile h5py

The log reports  "/MYOB/firedrake/src/h5py/h5py/api_compat.h(27): catastrophic error: cannot open source file "hdf5.h""

David:
...
One more piece of the puzzle for you: the activate script is part of Python venv. One of the first things that the install script does is
...
create a Firedrake venv into which Firedrake is installed. It is the creation of the venv which creates the activate script.
I see. Thanks.
...
I would reiterate what Lawrence says: installing by hand like this is very, very error-prone.
So too is not installing by hand it seems. At least for my setup.
...
In particular, installing by hand and then hoping that the update script will work sounds very optimistic indeed.
I share your pessimism. I hadn't expressed that hope, I just noted that that file hadn't been created.

I guess I could try building Python, as David suggested earlier. (Or going back to FEniCS, which installed without problems - but I do like quads :-)

Niall.

________________________________

From: Ham, David A <david.ham@imperial.ac.uk>
Sent: Thursday 22 August 2019 08:48
To: Lawrence Mitchell <wence@gmx.li>; Madden, Niall <niall.madden@nuigalway.ie>
Cc: Sagiyama, Koki <k.sagiyama@imperial.ac.uk>; firedrake <firedrake@imperial.ac.uk>
Subject: Re: [firedrake] Installing firedrake on a HPC system without package manager

Hi Niall,

One more piece of the puzzle for you: the activate script is part of Python venv. One of the first things that the install script does is create a Firedrake venv into which Firedrake is installed. It is the creation of the venv which creates the activate script.

I would reiterate what Lawrence says: installing by hand like this is very, very error-prone. It would almost certainly be better to run the install script. In particular, installing by hand and then hoping that the update script will work sounds very optimistic indeed.

Regards,

David

On 21/08/2019, 16:22, "Lawrence Mitchell" <wence@gmx.li> wrote:

    Hi Niall,

    On Wed, 21 Aug 2019 at 15:04, Madden, Niall <niall.madden@nuigalway.ie> wrote:
    >
    > Hi all,
    > I've returned to this task, and tried to follow David's suggestion
    > of building on a clean conda environment. This time, I used the intel compiler (mpiicc).
    >
    > Trying to follow the steps in the install script, we cloned
    > firedrake, ufl, fiat, PyOP2, COFFEE, loopy, petsc, supermesh, spatialindex, etc.
    >
    > Then we can successfully install all these. In particular, we can
    > configure/make/make install petsc
    >
    > Finally, we run, in firedrake,
    > python setup.py install
    > with LDSHARED, CFLAGS, CC, CXX, and prefix set appropriately.

    [...]

    > That is, if I try
    > echo "import firedrake" | python
    > I get
    >
    > Traceback (most recent call last):
    >   File "<stdin>", line 1, in <module>
    >   File "/MYOB/r/libs/firedrake/firedrake/__init__.py", line 5, in <module>
    >     if "PETSC_DIR" in os.environ and not config["options"]["honour_petsc_dir"]:
    > TypeError: 'NoneType' object is not subscriptable

    [...]

    > I note that, the "activate" script is not in firedrake/bin
    > Neither, as it happens, is  firedrake-update, but firedrake-clean, firedrake-install, and firedrake-zenodo are.

    This is an idiosyncrasy of the firedrake-install process.
    "firedrake-update" is actually just a copy of "firedrake-install" with
    a different name (so it knows it's in update mode).
    firedrake-{clean/install/zenodo} are created in the bin directory by
    setup.py, but firedrake-update is copied _by the firedrake-install
    process_. Hence it not existing because you did everything by hand.
    You can "fix" this by just copying firedrake-install to
    firedrake-update in the bin directory.

    The issue with the TypeError above is related. As well as creating the
    firedrake-update script, firedrake-install also writes a little
    configuration file that remembers the options you installed firedrake
    with (so that firedrake-update maintains them). Again, because you
    didn't run firedrake-install, this file doesn't exist.

    The relevant file to be created is `configuration.json` in the
    firedrake_configuration subdirectory of the firedrake source tree.
    Mine looks like:

    {"options": {"package_manager": false,
                 "minimal_petsc": false,
                 "mpicc": null,
                 "mpicxx": null,
                 "mpif90": null,
                 "mpiexec": null,
                 "disable_ssh": false,
                 "honour_petsc_dir": true,
                 "show_petsc_configure_options": false,
                 "slepc": true,
                 "packages": [],
                 "honour_pythonpath": false,
                 "petsc_int_type": "int32",
                 "cache_dir": "/home/vtdb72/Documents/work/src/firedrake/.cache"},
     "environment": {}, "additions": []}

    Editing this by hand is, however, rather error prone. Additionally,
    I'm not necessarily sure that firedrake-update will run as normal.
    Given you started with a clean conda environment, did you run into
    issues just running firedrake-install inside that?

    Thanks,

    Lawrence