Wait, what do you need the PYTHONPATH for? And why don't you load netCDF?
Yes, if you do that then python does run from within the Firedrake virtual environment...but this is essentially what we do when we submit a job to the scheduler:
#!/bin/sh
#PBS -d .
#PBS -q ptq
#PBS -l walltime=00:05:00
#PBS -A Research_Project-183035
#PBS -l nodes=1:ppn=16
#PBS -m e -M p.burns2@exeter.ac.uk
echo '<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>'
echo PBS_O_HOST = $PBS_O_HOST
echo PBS_ENVIRONMENT = $PBS_ENVIRONMENT
echo PBS_NODEFILE = $PBS_NODEFILE
echo '<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>'
module purge
#module load OpenMPI/2.0.0-GCC-4.8.5-torque
#module load Python/2.7.11-foss-2016a
module load OpenMPI/1.10.2-GCC-4.9.3-2.25
module load Python/3.5.1-foss-2016a
#module load netCDF/4.4.0-foss-2016a
. ~/firedrake-20171026/bin/activate
source ~/.bash_profile
#exec_address="./examples/boussinesq_2d_lab.py"
exec_address="./examples/sk_nonlinear.py"
echo -e 'Running Firedrake...'
mpirun \
-x FIREDRAKE_TSFC_KERNEL_CACHE_DIR \
-x PYOP2_CACHE_DIR \
-x LIBRARY_PATH \
-x LD_LIBRARY_PATH \
-x CPATH \
-x VIRTUAL_ENV \
-x PATH \
-x PKG_CONFIG_PATH \
-x PYTHONHOME \
-x PYTHONPATH \
-np 16 python $exec_address > log
So maybe the problem is that the python path is incorrect...
P
OK, now what if you load that Python module, and then activate the Firedrake venv, and then run
$ python
?
Confirmed - python does not run from within the virtual environment.
Note that for your second test to make any sense, you first would need to load a python module. So I did:
module load Python/3.5.1-foss-2016a
(foobar) [pb412@login02 ~]$ python
Python 3.5.1 (default, Jul 24 2017, 16:03:47)
[GCC 4.9.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
You don't even have a running Python in your venv! To confirm, can you just run
$ python
while still the Firedrake venv is still _active_?
And outside the Firedrake venv, try this:
$ python3 -mvenv foobar
$ . foobar/bin/activate
$ python
This is what happens when I execute the first line (from within virtual environment):
(firedrake-20171026) [pb412@login02 ~]$ pip uninstall netCDF4
/gpfs/ts0/home/pb412/firedrake-20171026/bin/python: error while loading shared libraries: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory
Paul
What if you try to reinstall netCDF4 from source inside the virtualenv? That is,
$ pip uninstall netCDF4
$ pip uninstall netCDF4
$ pip uninstall netCDF4
$ pip install --no-binary netCDF4 netCDF4
They are indeed syntax errors. Running Python 2 code under Python 3.
So the first important error message I can see in the log is:
-----------------------------------------------------------------------------------------------------
checking for library 'lmpe' ...
/gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -fPIC -c _configtest.c -o _configtest.o
/gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc _configtest.o -llmpe -o _configtest
/gpfs/ts0/shared/software/binutils/2.25-GCCcore-4.9.3/bin/ld.gold: error: cannot find -llmpe
collect2: error: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o
building 'mpe' dylib library
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/src
creating build/temp.linux-x86_64-3.5/src/lib-pmpi
/gpfs/ts0/shared/software/OpenMPI/1.10.2-GCC-4.9.3-2.25/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -fPIC -c src/lib-pmpi/mpe.c -o build/temp.linux-x86_64-3.5/src/lib-pmpi/mpe.o
creating build/lib.linux-x86_64-3.5/mpi4py/lib-pmpi
...
- To me it looks like the installer managed to work around the error but I am not certain.....?
Then further down the log there are a number of syntax-type errors. Here are the fist few:
-----------------------------------------------------------------------------------------------------
Removing source in /tmp/pip-build-b2dkectq/wrapt
*** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/all.py'...
File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/all.py", line 7
def func(): print 'yo'
^
SyntaxError: invalid syntax
*** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/invalid_encoding.py'...
File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/invalid_encoding.py", line 0
SyntaxError: unknown encoding: lala
*** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module.py'...
File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module.py", line 32
except ValueError, ex:
^
SyntaxError: invalid syntax
*** Error compiling '/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module2.py'...
File "/tmp/pip-build-b2dkectq/astroid/astroid/tests/testdata/python2/data/module2.py", line 78
exec 'c = 3'
^
SyntaxError: Missing parentheses in call to 'exec'
P
OK, great.
A quick google of the import error suggests this could be either a cython issue or a python version issue.
I wonder if it would be better to start from the errors that the usual firedrake install log gives?
[I'm at the end of my expertise on such matters now, by the way! What do others think?]
Jemma
By passing the flag '--install gusto' to the installer...
P
Hi Paul,
How did you install Gusto?
Jemma