Presumably we need to load all modules required by the Firedrake virtual environment prior to activating the virtual environment. I guess this is just a quirk of using clusters.
I believe that is correct, yes. ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 30 October 2017 09:07 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster I did try loading that netCDF4 module (in my submission script) prior to activating the Firedrake virtual environment, however, it then throws a version error at me. I have checked the list of available modules on the cluster and it does not include the netCDF4 version installed by the Firedrake installer (as below). Therefore possibly this is the problem. Presumably we need to load all modules required by the Firedrake virtual environment prior to activating the virtual environment. I guess this is just a quirk of using clusters. I am currently checking your other question - previously we found that the code only worked when we explicitly passed those environment variables to mpirun. ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk> Sent: 27 October 2017 13:33:56 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster Wait, what do you need the PYTHONPATH for? And why don't you load netCDF? ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 27 October 2017 13:29:41 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster Yes, if you do that then python does run from within the Firedrake virtual environment...but this is essentially what we do when we submit a job to the scheduler: #!/bin/sh #PBS -d . #PBS -q ptq #PBS -l walltime=00:05:00 #PBS -A Research_Project-183035 #PBS -l nodes=1:ppn=16 #PBS -m e -M p.burns2@exeter.ac.uk echo '<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>' echo PBS_O_HOST = $PBS_O_HOST echo PBS_ENVIRONMENT = $PBS_ENVIRONMENT echo PBS_NODEFILE = $PBS_NODEFILE echo '<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>' module purge #module load OpenMPI/2.0.0-GCC-4.8.5-torque #module load Python/2.7.11-foss-2016a module load OpenMPI/1.10.2-GCC-4.9.3-2.25 module load Python/3.5.1-foss-2016a #module load netCDF/4.4.0-foss-2016a . ~/firedrake-20171026/bin/activate source ~/.bash_profile #exec_address="./examples/boussinesq_2d_lab.py" exec_address="./examples/sk_nonlinear.py" echo -e 'Running Firedrake...' mpirun \ -x FIREDRAKE_TSFC_KERNEL_CACHE_DIR \ -x PYOP2_CACHE_DIR \ -x LIBRARY_PATH \ -x LD_LIBRARY_PATH \ -x CPATH \ -x VIRTUAL_ENV \ -x PATH \ -x PKG_CONFIG_PATH \ -x PYTHONHOME \ -x PYTHONPATH \ -np 16 python $exec_address > log So maybe the problem is that the python path is incorrect... P ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk> Sent: 27 October 2017 13:18:56 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster OK, now what if you load that Python module, and then activate the Firedrake venv, and then run $ python ? ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 27 October 2017 13:06:08 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster Confirmed - python does not run from within the virtual environment. Note that for your second test to make any sense, you first would need to load a python module. So I did: module load Python/3.5.1-foss-2016a python3 -mvenv foobar . foobar/bin/activate python After executing the last line python does run: (foobar) [pb412@login02 ~]$ python Python 3.5.1 (default, Jul 24 2017, 16:03:47) [GCC 4.9.3] on linux Type "help", "copyright", "credits" or "license" for more information.
________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk> Sent: 27 October 2017 12:49:59 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster You don't even have a running Python in your venv! To confirm, can you just run $ python while still the Firedrake venv is still _active_? And outside the Firedrake venv, try this: $ python3 -mvenv foobar $ . foobar/bin/activate $ python ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 27 October 2017 12:38:02 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster This is what happens when I execute the first line (from within virtual environment): (firedrake-20171026) [pb412@login02 ~]$ pip uninstall netCDF4 /gpfs/ts0/home/pb412/firedrake-20171026/bin/python: error while loading shared libraries: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory Paul ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk> Sent: 27 October 2017 12:19:54 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster What if you try to reinstall netCDF4 from source inside the virtualenv? That is, $ pip uninstall netCDF4 $ pip uninstall netCDF4 $ pip uninstall netCDF4 $ pip install --no-binary netCDF4 netCDF4 ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Homolya, Miklós <m.homolya14@imperial.ac.uk> Sent: 27 October 2017 11:59:39 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster They are indeed syntax errors. Running Python 2 code under Python 3. ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 27 October 2017 11:06:47 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster So the first important error message I can see in the log is: ----------------------------------------------------------------------------------------------------- checking for library 'lmpe' ... /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -fPIC -c _configtest.c -o _configtest.o /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc _configtest.o -llmpe -o _configtest /gpfs/ts0/shared/software/binutils/2.25-GCCcore-4.9.3/bin/ld.gold: error: cannot find -llmpe collect2: error: ld returned 1 exit status failure. removing: _configtest.c _configtest.o building 'mpe' dylib library creating build/temp.linux-x86_64-3.5 creating build/temp.linux-x86_64-3.5/src creating build/temp.linux-x86_64-3.5/src/lib-pmpi /gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -fPIC -c src/lib-pmpi/mpe.c -o build/temp.linux-x86_64-3.5/src/lib-pmpi/mpe.o creating build/lib.linux-x86_64-3.5/mpi4py/lib-pmpi ... ----------------------------------------------------------------------------------------------------- - To me it looks like the installer managed to work around the error but I am not certain.....? Then further down the log there are a number of syntax-type errors. Here are the fist few: ----------------------------------------------------------------------------------------------------- Removing source in /tmp/pip-build-b2dkectq/wrapt *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/all.py'... File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/all.py", line 7 def func(): print 'yo' ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/invalid_encoding.py'... File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/invalid_encoding.py", line 0 SyntaxError: unknown encoding: lala *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module.py'... File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module.py", line 32 except ValueError, ex: ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module2.py'... File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module2.py", line 78 exec 'c = 3' ^ SyntaxError: Missing parentheses in call to 'exec' ----------------------------------------------------------------------------------------------------- P ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 27 October 2017 10:39:31 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster Previously I did search the firedrake-install.log file for pattern 'netcdf4'...this is the last occurrence: Looking up "https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25e..." in the cache No cache entry available "GET /packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25edb3a144af71f88a2dfcb69a/netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_64.whl HTTP/1.1" 200 6163771 Downloading netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_64.whl (6.2MB) Downloading from URL https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25e... (from https://pypi.python.org/simple/netcdf4/) Updating cache with response from "https://pypi.python.org/packages/c0/06/c59f37661ede4f9ae1af893e08f9c214bc25e..." Caching due to etag Requirement already satisfied: numpy>=1.7 in /gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages (from netCDF4->-r gusto/requirements.txt (line 1)) Installing collected packages: netCDF4 It looks like it struggled to find a compatible version of netcdf4 but in the end was successful and installed this version of netcdf4: netCDF4-1.3.0-cp35-cp35m-manylinux1_x86_6 I will have another look at the install log and search for errors. See attached the install log - I am not an expert in this stuff. P ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Shipton, Jemma <j.shipton@imperial.ac.uk> Sent: 26 October 2017 18:06:25 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster OK, great. A quick google of the import error suggests this could be either a cython issue or a python version issue. I wonder if it would be better to start from the errors that the usual firedrake install log gives? [I'm at the end of my expertise on such matters now, by the way! What do others think?] Jemma ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 26 October 2017 17:55:20 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster By passing the flag '--install gusto' to the installer... P ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Shipton, Jemma <j.shipton@imperial.ac.uk> Sent: 26 October 2017 15:14:50 To: firedrake Subject: Re: [firedrake] Firedrake installation on cluster Hi Paul, How did you install Gusto? Jemma ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Burns, Paul <P.Burns2@exeter.ac.uk> Sent: 26 October 2017 15:10:14 To: firedrake Subject: [firedrake] Firedrake installation on cluster Dear Sir/Madam, I am trying to get Firedrake up and running on the Isca cluster at Exeter. The standard install method (i.e., executing 'python3 firedrake-install') does not work, however, we have managed to install Firedrake by developing an install script that loads modules manually, that sets a number of petsc options, and that passes a number of extra options to firedrake-install. However, when I try and run a simulation on Gusto I get the following error: Traceback (most recent call last): File "./examples/boussinesq_2d_lab.py", line 1, in <module> from gusto import * File "/gpfs/ts0/home/pb412/firedrake-20171026/src/gusto/gusto/__init__.py", line 11, in <module> from gusto.state import * # noqa File "/gpfs/ts0/home/pb412/firedrake-20171026/src/gusto/gusto/state.py", line 3, in <module> from netCDF4 import Dataset File "/gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages/netCDF4/__init__.py", line 3, in <module> from ._netCDF4 import * ImportError: /gpfs/ts0/home/pb412/firedrake-20171026/lib/python3.5/site-packages/netCDF4/_netCDF4.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _PyThreadState_UncheckedGet So it looks like there is some problem with the netCDF4 library. Please advise. Regards, Paul