Dear all, just to ley you know that it is working now, i.e. I can run the helmholtz solver on 4 processes on ARCHER. I needed the local_par-loop branch from the latest version of PyOP2 and also the multigrid branch from the latest firedrake, so I pulled these two into my work directory. I then added export PYTHONPATH=$WORK/PyOP2:${PYTHONPATH} export PYTHONPATH=$WORK/firedrake:${PYTHONPATH} to my job script (attached). However, despite that, python seems to load the pre-installed PyOP2 module first, below is the output of sys.path. I could fix this by adding import sys sys.path.insert(0,'/work/n02/n02/eike/PyOP2') [...] at the top of my main python script. Any ideas why I had to do this? Thanks, Eike ['/fs2/n02/n02/eike/git_workspace/firedrake-helmholtzsolver/source', '/fs2/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages/decorator-3.4.0-py2.7.egg', '/work/y07/y07/fdrake/PyOP2/lib/python2.7/site-packages/PyOP2-0.11.0_99_g54eb9ea_dirty-py2.7-linux-x86_64.egg', '/work/y07/y07/fdrake/ffc/lib/python2.7/site-packages/FFC-1.4.0_-py2.7-linux-x86_64.egg', '/work/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages/decorator-3.4.0-py2.7.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/python_hostlist-1.14-py2.7.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/pymongo-2.7.2-py2.7-linux-x86_64.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/GridDataFormats-0.2.4-py2.7.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/MDAnalysis-0.8.1-py2.7-linux-x86_64.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/python_hostlist-1.14-py2.7.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/pymongo-2.7.2-py2.7-linux-x86_64.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/radical.utils-0.7.7-py2.7.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/saga_python-0.18-py2.7.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/GridDataFormats-0.2.4-py2.7.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/apache_libcloud-0.15.1-py2.7.egg', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg', '/work/n02/n02/eike/firedrake', '/work/n02/n02/eike/PyOP2', '/work/n02/n02/eike/firedrake-bench', '/work/n02/n02/eike/pybench', '/work/y07/y07/fdrake/firedrake/lib/python2.7/site-packages', '/work/y07/y07/fdrake/PyOP2/lib/python2.7/site-packages', '/work/y07/y07/fdrake/ufl/lib/python2.7/site-packages', '/work/y07/y07/fdrake/scientificpython/lib/python2.7/site-packages', '/work/y07/y07/fdrake/psutil/lib/python2.7/site-packages', '/work/y07/y07/fdrake/mpi4py/lib/python2.7/site-packages', '/work/y07/y07/fdrake/instant/lib/python2.7/site-packages', '/work/y07/y07/fdrake/fiat/lib/python2.7/site-packages', '/work/y07/y07/fdrake/ffc/lib/python2.7/site-packages', '/work/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages', '/work/y07/y07/fdrake/petsc/arch-linux2-cxx-opt/lib/python2.7/site-packages', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7', '/opt/cray/sdb/1.0-1.0501.48084.4.48.ari/lib64/py', '/work/y07/y07/cse/anaconda/1.9.2/lib/python27.zip', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/plat-linux2', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-tk', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-old', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-dynload', '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/PIL', '/opt/cray/sdb/1.0-1.0501.48084.4.48.ari/lib64/py'] On 15/09/14 09:37, Florian Rathgeber wrote:
On 15/09/14 09:20, Eike Mueller wrote:
Hi Florian,
I'm having another go at this now.
export PYTHONPATH=${FDRAKE_DIR}/petsc4py/lib/python2.7/site-packages:${PYTHONPATH} export PYTHONPATH=${FDRAKE_DIR}/ufl/lib/python2.7/site-packages/:${PYTHONPATH} export PYTHONPATH=${FDRAKE_DIR}/fiat/lib/python2.7/site-packages/:${PYTHONPATH} export PYTHONPATH=${FDRAKE_DIR}/ffc/lib/python2.7/site-packages/:${PYTHONPATH}
I added this to the template file on the wiki. This *should* be added by loading fdrake-python-env
* The https cloning issue I believe is resolved, at least I can't reproduce it anymore. All seems to work fine for me now.
* The PyOP2 installation issue related to NumPy is a bug [1] in setuptools 2.2 which comes with anaconda. For now, the best thing to do is to remove/comment line 138 in PyOP2/setup.py which declares the setup dependency on NumPy.
I commented out the line 'setup_requires=setup_requires,' in setup.py and the PyOP2 installation goes through without problems.
* The anaconda/environment modules conflict is resolved: I have removed ANACONDA_LIB from the LD_LIBRARY_PATH and added it to LD_RUN_PATH instead. This means Python extension modules get an RPATH to that directory baked in and therefore can find the right libpython2.7.so without any LD_LIBRARY_PATH being set (which is preferable anyway).
Works for me now, I can run the module command even after loading the firedrake environment.
Unfortunately, Firedrake's setup.py manually sets an RPATH for PETSc and therefore the LD_RUN_PATH is ignored. This patch is required:
diff --git a/setup.py b/setup.py index 00838f8..350d51d 100644 --- a/setup.py +++ b/setup.py @@ -70,6 +70,7 @@ setup(name='firedrake', include_dirs=include_dirs, libraries=["petsc"], extra_link_args=["-L%s/lib" % d for d in petsc_dirs] + - ["-Wl,-rpath,%s/lib" % d for d in petsc_dirs]), + ["-Wl,-rpath,%s/lib" % d for d in petsc_dirs] + + ["-Wl,-rpath,%s/lib" % sys.prefix]), Extension('evtk.cevtk', evtk_sources, include_dirs=[np.get_include()])])
Can you send me that patch as a separate text file, please? When I copied it out of your email it didn't work. I have pushed this to firedrake master.
Still something not quite right with the python. When I run a 'Hello World!' test, i.e.
import sys print 'Hello World!' sys.exit(0)
I still get
python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory That works for me. Not sure why this would fail since you're not loading an extension module. Is this on the login node with anaconda?
but I guess this is due to the unresolved issue with anaconda that you reported to the helpdesk, so I will wait for what comes back from them. Indeed, as reported to helpdesk, it was premature thinking the LD_LIBRARY_PATH was no longer required. I'm currently stuck on mpi4py even though all the RPATHs are correctly set afaict. For now you'll need to add $ANANCONDA_LIBS to LD_LIBRARY_PATH in your firedrake.tpl
Florian
Thanks,
Eike
For FFC it's more tricky, still trying to figure out what's needed.
Keep us posted on how things are progressing!
Florian
I will go ahead and edit the wiki.
Eike
On 8 Sep 2014, at 15:53, Eike Mueller <E.Mueller@bath.ac.uk <mailto:E.Mueller@bath.ac.uk> <mailto:E.Mueller@bath.ac.uk>> wrote:
Hi Florian,
thanks a lot for your reply and for updating the wiki, I got a bit further now.
You're exactly right again: it seems that anaconda contains a version of the TCL libraries which are incompatible with those use by the environment modules. I'm not sure how to work around this, since we need the anaconda libs (as you figured out later) - will open a ticket. ok, I will use the environment script as it is and not worry about the fact that the anaconda library breaks the module command at the moment. Let's see if the support can figure something out.
I've changed the wiki page to clone via https, thanks! It works with the updated instructions on the webpage. Strangely, if I clone a repository into a directory, delete this directory and then try to clone again with exactly the same command, nothing happens.
This seems to happen because (for reasons I can't figure out) setuptools doesn't detect that NumPy is already installed and builds it from source. The primary cause of the error you're seeing is this: https://stackoverflow.com/a/21621493 I will ignore this then, and just call the setup.py script twice in the PyOP2 directory.
This is because you've removed the anaconda lib path, which also contains the Python libraries themselves, from LD_LIBRARY_PATH I tried a simpler hello-world script where I swap python -> anaconda and run a python script. This does work. Still trying to run the poisson example from the benchmark repository, but the ARCHER queues seem to be terribly slow, even if I just run on one node on the debug queue :-(
Absolutely, feel free to edit the wiki if you find more errors or things that are unclear.
I think I already have access to the repository, so will edit if necessary.
Thanks a lot,
Eike
Florian
> Cheers, > > Eike > > *** content of firedrake.env *** > > module swap PrgEnv-cray PrgEnv-gnu > module unload python > module add anaconda > > # Add fdrake module path > export FDRAKE_DIR=/work/y07/y07/fdrake > module use "$FDRAKE_DIR/modules" > > # Load build environment > module load fdrake-build-env > module load fdrake-python-env > > LD_LIBRARY_PATH=`echo $LD_LIBRARY_PATH | sed > 's/\/work\/y07\/y07\/cse\/anaconda\/1.9.2\/lib//g'` > > export > PYTHONPATH=$WORK/firedrake-bench:$WORK/firedrake-bench/pybench:$WORK/firedrake:$WORK/PyOP2:$PYTHONPATH
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
-- Dr Eike Hermann Mueller Research Associate (PostDoc) Department of Mathematical Sciences University of Bath Bath BA2 7AY, United Kingdom +44 1225 38 5803 e.mueller@bath.ac.uk http://people.bath.ac.uk/em459/