OK, now what if you load that Python module, and then activate the Firedrake venv, and then run $ python ? ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 27 October 2017 13:06:08 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster Confirmed - python does not run from within the virtual environment. Note that for your second test to make any sense, you first would need to load a python module. So I did: module load Python/3.5.1-foss-2016a python3 -mvenv foobar . foobar/bin/activate python After executing the last line python does run: (foobar) [pb412@login02 ~]$ python Python 3.5.1 (default, Jul 24 2017, 16:03:47) [GCC 4.9.3] on linux Type "help", "copyright", "credits" or "license" for more information.
________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk> Sent: 27 October 2017 12:49:59 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster You don't even have a running Python in your venv! To confirm, can you just run $ python while still the Firedrake venv is still _active_? And outside the Firedrake venv, try this: $ python3 -mvenv foobar $ . foobar/bin/activate $ python ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 27 October 2017 12:38:02 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster This is what happens when I execute the first line (from within virtual environment): (firedrake-20171026) [pb412@login02 ~]$ pip uninstall netCDF4 /gpfs/ts0/home/pb412/firedrake-20171026/bin/python: error while loading shared libraries: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory Paul ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk> Sent: 27 October 2017 12:19:54 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster What if you try to reinstall netCDF4 from source inside the virtualenv? That is, $ pip uninstall netCDF4 $ pip uninstall netCDF4 $ pip uninstall netCDF4 $ pip install --no-binary netCDF4 netCDF4 ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk> Sent: 27 October 2017 11:59:39 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster They are indeed syntax errors. Running Python 2 code under Python 3. ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 27 October 2017 11:06:47 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster So the first important error message I can see in the log is: ----------------------------------------------------------------------------------------------------- checking for library 'lmpe' ... /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -fPIC -c _configtest.c -o _configtest.o /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc _configtest.o -llmpe -o _configtest /gpfs/ts0/shared/software/binutils/2.25-GCCcore-4.9.3/bin/ld.gold: error: cannot find -llmpe collect2: error: ld returned 1 exit status failure. removing: _configtest.c _configtest.o building 'mpe' dylib library creating build/temp.linux-x86_64-3.5 creating build/temp.linux-x86_64-3.5/src creating build/temp.linux-x86_64-3.5/src/lib-pmpi /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -fPIC -c src/lib-pmpi/mpe.c -o build/temp.linux-x86_64-3.5/src/lib-pmpi/mpe.o creating build/lib.linux-x86_64-3.5/mpi4py/lib-pmpi ... ----------------------------------------------------------------------------------------------------- - To me it looks like the installer managed to work around the error but I am not certain.....? Then further down the log there are a number of syntax-type errors. Here are the fist few: ----------------------------------------------------------------------------------------------------- Removing source in /tmp/pip-build-b2dkectq/wrapt *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/all.py'... File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/all.py", line 7 def func(): print 'yo' ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/invalid_encoding.py'... File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/invalid_encoding.py", line 0 SyntaxError: unknown encoding: lala *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module.py'... File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module.py", line 32 except ValueError, ex: ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module2.py'... File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module2.py", line 78 exec 'c = 3' ^ SyntaxError: Missing parentheses in call to 'exec' ----------------------------------------------------------------------------------------------------- P ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 27 October 2017 10:39:31 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster Previously I did search the firedrake-install.log file for pattern 'netcdf4'...this is the last occurrence: Looking up "https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25e..." in the cache No cache entry available "GET /packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25edb3a144af71f88a2dfcb69a/netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_64.whl HTTP/1.1" 200 6163771 Downloading netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_64.whl (6.2MB) Downloading from URL https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25e... (from https://pypi.python.org/simple/netcdf4/) Updating cache with response from "https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25e..." Caching due to etag Requirement already satisfied: numpy>=1.7 in /gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages (from netCDF4->-r gusto/requirements.txt (line 1)) Installing collected packages: netCDF4 It looks like it struggled to find a compatible version of netcdf4 but in the end was successful and installed this version of netcdf4: netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_6 I will have another look at the install log and search for errors. See attached the install log - I am not an expert in this stuff. P ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Shipton, Jemma <j.shipton@imperial.ac.uk> Sent: 26 October 2017 18:06:25 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster OK, great. A quick google of the import error suggests this could be either a cython issue or a python version issue. I wonder if it would be better to start from the errors that the usual firedrake install log gives? [I'm at the end of my expertise on such matters now, by the way! What do others think?] Jemma ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 26 October 2017 17:55:20 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster By passing the flag '--install gusto' to the installer... P ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Shipton, Jemma <j.shipton@imperial.ac.uk> Sent: 26 October 2017 15:14:50 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster Hi Paul, How did you install Gusto? Jemma ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 26 October 2017 15:10:14 To: firedrake Subject: [firedrake] Firedrake installation on cluster Dear Sir/Madam, I am trying to get Firedrake up and running on the Isca cluster at Exeter. The standard install method (i.e., executing 'python3 firedrake-install') does not work, however, we have managed to install Firedrake by developing an install script that loads modules manually, that sets a number of petsc options, and that passes a number of extra options to firedrake-install. However, when I try and run a simulation on Gusto I get the following error: Traceback (most recent call last): File "./examples/boussinesq_2d_lab.py", line 1, in <module> from gusto import * File "/gpfs/ts0/home/pb412/firedrake-20171026/src/gusto/gusto/__init__.py", line 11, in <module> from gusto.state import * # noqa File "/gpfs/ts0/home/pb412/firedrake-20171026/src/gusto/gusto/state.py", line 3, in <module> from netCDF4 import Dataset File "/gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages/netCDF4/__init__.py", line 3, in <module> from ._netCDF4 import * ImportError: /gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages/netCDF4/_netCDF4.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _PyThreadState_UncheckedGet So it looks like there is some problem with the netCDF4 library. Please advise. Regards, Paul