Re: [firedrake] Firedrake installation on cluster

30 Oct 2017

      ...
They think that as long as we enable the Firedrake Python virtual environment to 'fire up', by loading the relevant Python module, before trying to activate the virtual environment, then the correctly activated virtual environment should then take care of the rest of the required software.
I'm not sure I can fully believe that. I mean, the Firedrake virtual environment is self-contained with respect to Python, so it should take care of all the Python dependencies. We also take care of PETSc, and some other compiled libraries, but some compiled libraries are considered "system". For example, we rely on whatever compilers and MPI implementation the host system provides, we only install Cython and mpi4py ourselves in the Python virtualenv.

I don't know what the deal with NetCDF is, perhaps some gusto people know the answer: is compiled NetCDF expected to be present as a pre-installed library, or does Gusto have an installer that takes care of that?

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 30 October 2017 12:22:26
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

Hi,

So I discussed this with the Isca support team.

They think that as long as we enable the Firedrake Python virtual environment to 'fire up', by loading the relevant Python module, before trying to activate the virtual environment, then the correctly activated virtual environment should then take care of the rest of the required software.  The exception is that we also need to load an OpenMPI module, as in the submission script.  So they don't think we need to load a netCDF4 module - they think this should be sorted out by the Firedrake virtual environment.

To answer the other question more completely - we found that we had to pass the environment variables to mpirun because otherwise the slave MPI jobs were missing the virtual environment and using the system Python instead.

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk>
Sent: 30 October 2017 10:22:03
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster
...
Presumably we need to load all modules required by the Firedrake virtual environment prior to activating the virtual environment.  I guess this is just a quirk of using clusters.
I believe that is correct, yes.

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 30 October 2017 09:07
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

I did try loading that netCDF4 module (in my submission script) prior to activating the Firedrake virtual environment, however, it then throws a version error at me.

I have checked the list of available modules on the cluster and it does not include the netCDF4 version installed by the Firedrake installer (as below).  Therefore possibly this is the problem.

Presumably we need to load all modules required by the Firedrake virtual environment prior to activating the virtual environment.  I guess this is just a quirk of using clusters.

I am currently checking your other question - previously we found that the code only worked when we explicitly passed those environment variables to mpirun.

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk>
Sent: 27 October 2017 13:33:56
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

Wait, what do you need the PYTHONPATH for? And why don't you load netCDF?

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 27 October 2017 13:29:41
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

Yes, if you do that then python does run from within the Firedrake virtual environment...but this is essentially what we do when we submit a job to the scheduler:

#!/bin/sh

#PBS -d .

#PBS -q ptq

#PBS -l walltime=00:05:00

#PBS -A Research_Project-183035

#PBS -l nodes=1:ppn=16

#PBS -m e -M p.burns2@exeter.ac.uk

echo '<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>'

echo PBS_O_HOST      = $PBS_O_HOST

echo PBS_ENVIRONMENT = $PBS_ENVIRONMENT

echo PBS_NODEFILE    = $PBS_NODEFILE

echo '<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>'

module purge

#module load OpenMPI/2.0.0-GCC-4.8.5-torque

#module load Python/2.7.11-foss-2016a

module load OpenMPI/1.10.2-GCC-4.9.3-2.25

module load Python/3.5.1-foss-2016a

#module load netCDF/4.4.0-foss-2016a

. ~/firedrake-20171026/bin/activate

source ~/.bash_profile

#exec_address="./examples/boussinesq_2d_lab.py"

exec_address="./examples/sk_nonlinear.py"

echo -e 'Running Firedrake...'

mpirun \

-x FIREDRAKE_TSFC_KERNEL_CACHE_DIR \

-x PYOP2_CACHE_DIR \

-x LIBRARY_PATH \

-x LD_LIBRARY_PATH \

-x CPATH \

-x VIRTUAL_ENV \

-x PATH \

-x PKG_CONFIG_PATH \

-x PYTHONHOME \

-x PYTHONPATH \

-np 16 python $exec_address > log

So maybe the problem is that the python path is incorrect...

P

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk>
Sent: 27 October 2017 13:18:56
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

OK, now what if you load that Python module, and then activate the Firedrake venv, and then run

$ python

?

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 27 October 2017 13:06:08
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

Confirmed - python does not run from within the virtual environment.

Note that for your second test to make any sense, you first would need to load a python module.  So I did:

module load Python/3.5.1-foss-2016a

python3 -mvenv foobar
. foobar/bin/activate
python

After executing the last line python does run:

(foobar) [pb412@login02 ~]$ python

Python 3.5.1 (default, Jul 24 2017, 16:03:47)

[GCC 4.9.3] on linux

Type "help", "copyright", "credits" or "license" for more information.
...
...
...
________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk>
Sent: 27 October 2017 12:49:59
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

You don't even have a running Python in your venv! To confirm, can you just run

$ python

while still the Firedrake venv is still _active_?

And outside the Firedrake venv, try this:

$ python3 -mvenv foobar

$ . foobar/bin/activate

$ python

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 27 October 2017 12:38:02
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

This is what happens when I execute the first line (from within virtual environment):

(firedrake-20171026) [pb412@login02 ~]$ pip uninstall netCDF4

/gpfs/ts0/home/pb412/firedrake-20171026/bin/python: error while loading shared libraries: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory

Paul

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk>
Sent: 27 October 2017 12:19:54
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

What if you try to reinstall netCDF4 from source inside the virtualenv? That is,

$ pip uninstall netCDF4

$ pip uninstall netCDF4

$ pip uninstall netCDF4

$ pip install --no-binary netCDF4 netCDF4

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk>
Sent: 27 October 2017 11:59:39
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

They are indeed syntax errors. Running Python 2 code under Python 3.

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 27 October 2017 11:06:47
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

So the first important error message I can see in the log is:

-----------------------------------------------------------------------------------------------------

   checking for library 'lmpe' ...

    /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -fPIC -c _configtest.c -o _configtest.o

    /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc _configtest.o -llmpe -o _configtest

    /gpfs/ts0/shared/software/binutils/2.25-GCCcore-4.9.3/bin/ld.gold: error: cannot find -llmpe

    collect2: error: ld returned 1 exit status

    failure.

    removing: _configtest.c _configtest.o

    building 'mpe' dylib library

    creating build/temp.linux-x86_64-3.5

    creating build/temp.linux-x86_64-3.5/src

    creating build/temp.linux-x86_64-3.5/src/lib-pmpi

    /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -fPIC -c src/lib-pmpi/mpe.c -o build/temp.linux-x86_64-3.5/src/lib-pmpi/mpe.o

    creating build/lib.linux-x86_64-3.5/mpi4py/lib-pmpi

...

-----------------------------------------------------------------------------------------------------

- To me it looks like the installer managed to work around the error but I am not certain.....?

Then further down the log there are a number of syntax-type errors.  Here are the fist few:

-----------------------------------------------------------------------------------------------------

 Removing source in /tmp/pip-build-b2dkectq/wrapt

  *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/all.py'...

    File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/all.py", line 7

      def func(): print 'yo'

                           ^

  SyntaxError: invalid syntax

  *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/invalid_encoding.py'...

    File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/invalid_encoding.py", line 0

  SyntaxError: unknown encoding: lala

  *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module.py'...

    File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module.py", line 32

      except ValueError, ex:

                       ^

  SyntaxError: invalid syntax

  *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module2.py'...

    File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module2.py", line 78

      exec 'c = 3'

                 ^

  SyntaxError: Missing parentheses in call to 'exec'

-----------------------------------------------------------------------------------------------------

P

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 27 October 2017 10:39:31
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

Previously I did search the firedrake-install.log file for pattern 'netcdf4'...this is the last occurrence:

  Looking up "https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25e..." in the cache
  No cache entry available
  "GET /packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25edb3a144af71f88a2dfcb69a/netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_64.whl HTTP/1.1" 200 6163771
  Downloading netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_64.whl (6.2MB)
  Downloading from URL https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25e... (from https://pypi.python.org/simple/netcdf4/)
  Updating cache with response from "https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25e..."
  Caching due to etag
Requirement already satisfied: numpy>=1.7 in /gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages (from netCDF4->-r gusto/requirements.txt (line 1))

Installing collected packages: netCDF4

It looks like it struggled to find a compatible version of netcdf4 but in the end was successful and installed this version of netcdf4:

netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_6

I will have another look at the install log and search for errors.  See attached the install log - I am not an expert in this stuff.

P

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Shipton, Jemma <j.shipton@imperial.ac.uk>
Sent: 26 October 2017 18:06:25
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

OK, great.

A quick google of the import error suggests this could be either a cython issue or a python version issue.

I wonder if it would be better to start from the errors that the usual firedrake install log gives?

[I'm at the end of my expertise on such matters now, by the way! What do others think?]

Jemma

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 26 October 2017 17:55:20
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

By passing the flag '--install gusto' to the installer...

P

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Shipton, Jemma <j.shipton@imperial.ac.uk>
Sent: 26 October 2017 15:14:50
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster

Hi Paul,

How did you install Gusto?

Jemma

________________________________
From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 26 October 2017 15:10:14
To: firedrake
Subject: [firedrake] Firedrake installation on cluster

Dear Sir/Madam,

I am trying to get Firedrake up and running on the Isca cluster at Exeter.  The standard install method (i.e., executing 'python3 firedrake-install') does not work, however, we have managed to install Firedrake by developing an install script that loads modules manually, that sets a number of petsc options, and that passes a number of extra options to firedrake-install.

However, when I try and run a simulation on Gusto I get the following error:

Traceback (most recent call last):
  File "./examples/boussinesq_2d_lab.py", line 1, in <module>
    from gusto import *
  File "/gpfs/ts0/home/pb412/firedrake-20171026/src/gusto/gusto/__init__.py", line 11, in <module>
    from gusto.state import *           # noqa
  File "/gpfs/ts0/home/pb412/firedrake-20171026/src/gusto/gusto/state.py", line 3, in <module>
    from netCDF4 import Dataset
  File "/gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages/netCDF4/__init__.py", line 3, in <module>
    from ._netCDF4 import *

ImportError: /gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages/netCDF4/_netCDF4.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _PyThreadState_UncheckedGet

So it looks like there is some problem with the netCDF4 library.

Please advise.

Regards,
Paul