This is what happens when I execute the first line (from within virtual environment):


(firedrake-20171026) [pb412@login02 ~]$ pip uninstall netCDF4

/gpfs/ts0/home/pb412/firedrake-20171026/bin/python: error while loading shared libraries: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory


Paul

From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk>
Sent: 27 October 2017 12:19:54
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster
 

What if you try to reinstall netCDF4 from source inside the virtualenv? That is,


$ pip uninstall netCDF4

$ pip uninstall netCDF4

$ pip uninstall netCDF4

$ pip install --no-binary netCDF4 netCDF4


From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk>
Sent: 27 October 2017 11:59:39
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster
 

They are indeed syntax errors. Running Python 2 code under Python 3.

From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 27 October 2017 11:06:47
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster
 

So the first important error message I can see in the log is:


-----------------------------------------------------------------------------------------------------

   checking for library 'lmpe' ...

    /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -fPIC -c _configtest.c -o _configtest.o

    /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc _configtest.o -llmpe -o _configtest

    /gpfs/ts0/shared/software/binutils/2.25-GCCcore-4.9.3/bin/ld.gold: error: cannot find -llmpe

    collect2: error: ld returned 1 exit status

    failure.

    removing: _configtest.c _configtest.o

    building 'mpe' dylib library

    creating build/temp.linux-x86_64-3.5

    creating build/temp.linux-x86_64-3.5/src

    creating build/temp.linux-x86_64-3.5/src/lib-pmpi

    /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -fPIC -c src/lib-pmpi/mpe.c -o build/temp.linux-x86_64-3.5/src/lib-pmpi/mpe.o

    creating build/lib.linux-x86_64-3.5/mpi4py/lib-pmpi

...

-----------------------------------------------------------------------------------------------------

- To me it looks like the installer managed to work around the error but I am not certain.....?



Then further down the log there are a number of syntax-type errors.  Here are the fist few:

-----------------------------------------------------------------------------------------------------

 Removing source in /tmp/pip-build-b2dkectq/wrapt


  *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/all.py'...

    File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/all.py", line 7

      def func(): print 'yo'

                           ^

  SyntaxError: invalid syntax

  

  *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/invalid_encoding.py'...

    File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/invalid_encoding.py", line 0

  SyntaxError: unknown encoding: lala

  

  *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module.py'...

    File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module.py", line 32

      except ValueError, ex:

                       ^

  SyntaxError: invalid syntax

  

  *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module2.py'...

    File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module2.py", line 78

      exec 'c = 3'

                 ^

  SyntaxError: Missing parentheses in call to 'exec'

-----------------------------------------------------------------------------------------------------





P




From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 27 October 2017 10:39:31
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster
 
Previously I did search the firedrake-install.log file for pattern 'netcdf4'...this is the last occurrence:

  Looking up "https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25edb3a144af71f88a2dfcb69a/netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_64.whl" in the cache
  No cache entry available
  "GET /packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25edb3a144af71f88a2dfcb69a/netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_64.whl HTTP/1.1" 200 6163771
  Downloading netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_64.whl (6.2MB)
  Downloading from URL https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25edb3a144af71f88a2dfcb69a/netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_64.whl#md5=d4660a6dea3062578fc9cfec6b243949 (from https://pypi.python.org/simple/netcdf4/)
  Updating cache with response from "https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25edb3a144af71f88a2dfcb69a/netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_64.whl"
  Caching due to etag
Requirement already satisfied: numpy>=1.7 in /gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages (from netCDF4->-r gusto/requirements.txt (line 1))
 
Installing collected packages: netCDF4


It looks like it struggled to find a compatible version of netcdf4 but in the end was successful and installed this version of netcdf4:

netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_6

I will have another look at the install log and search for errors.  See attached the install log - I am not an expert in this stuff.


P

From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Shipton, Jemma <j.shipton@imperial.ac.uk>
Sent: 26 October 2017 18:06:25
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster
 

OK, great.


A quick google of the import error suggests this could be either a cython issue or a python version issue.


I wonder if it would be better to start from the errors that the usual firedrake install log gives?


[I'm at the end of my expertise on such matters now, by the way! What do others think?]


Jemma

From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 26 October 2017 17:55:20
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster
 

By passing the flag '--install gusto' to the installer...


P

From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Shipton, Jemma <j.shipton@imperial.ac.uk>
Sent: 26 October 2017 15:14:50
To: firedrake
Subject: Re: [firedrake] Firedrake installation on cluster
 

Hi Paul,


How did you install Gusto?


Jemma

From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk>
Sent: 26 October 2017 15:10:14
To: firedrake
Subject: [firedrake] Firedrake installation on cluster
 
Dear Sir/Madam,

I am trying to get Firedrake up and running on the Isca cluster at Exeter.  The standard install method (i.e., executing 'python3 firedrake-install') does not work, however, we have managed to install Firedrake by developing an install script that loads modules manually, that sets a number of petsc options, and that passes a number of extra options to firedrake-install.

However, when I try and run a simulation on Gusto I get the following error: 

Traceback (most recent call last):
  File "./examples/boussinesq_2d_lab.py", line 1, in <module>
    from gusto import *
  File "/gpfs/ts0/home/pb412/firedrake-20171026/src/gusto/gusto/__init__.py", line 11, in <module>
    from gusto.state import *           # noqa
  File "/gpfs/ts0/home/pb412/firedrake-20171026/src/gusto/gusto/state.py", line 3, in <module>
    from netCDF4 import Dataset
  File "/gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages/netCDF4/__init__.py", line 3, in <module>
    from ._netCDF4 import *
 
ImportError: /gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages/netCDF4/_netCDF4.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _PyThreadState_UncheckedGet


So it looks like there is some problem with the netCDF4 library.

Please advise.

Regards,
Paul