On 29/09/14 11:08, Eike Mueller wrote:

Dear all,

just to ley you know that it is working now, i.e. I can run the
helmholtz solver on 4 processes on ARCHER.

I needed the local_par-loop branch from the latest version of PyOP2 and
also the multigrid branch from the latest firedrake, so I pulled these
two into my work directory.
I then added

export PYTHONPATH=$WORK/PyOP2:${PYTHONPATH}
export PYTHONPATH=$WORK/firedrake:${PYTHONPATH}

to my job script (attached). However, despite that, python seems to load
the pre-installed PyOP2 module first, below is the output of sys.path.

I could fix this by adding

import sys
sys.path.insert(0,'/work/n02/n02/eike/PyOP2')
[...]

at the top of my main python script. Any ideas why I had to do this?

The reason for this is a "feature" of distutils, which generates a file
easy_install.pth containing paths to eggs installed using distutils
which are pushed to the *beginning* of sys.path even before entries in
the $PYTHONPATH. I have fixed this now, so you no longer need to fiddle
with sys.path in your script.

Florian

Thanks,

Eike


['/fs2/n02/n02/eike/git_workspace/firedrake-helmholtzsolver/source',
'/fs2/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages/decorator-3.4.0-py2.7.egg',
'/work/y07/y07/fdrake/PyOP2/lib/python2.7/site-packages/PyOP2-0.11.0_99_g54eb9ea_dirty-py2.7-linux-x86_64.egg',
'/work/y07/y07/fdrake/ffc/lib/python2.7/site-packages/FFC-1.4.0_-py2.7-linux-x86_64.egg',
'/work/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages/decorator-3.4.0-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/python_hostlist-1.14-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/pymongo-2.7.2-py2.7-linux-x86_64.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/GridDataFormats-0.2.4-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/MDAnalysis-0.8.1-py2.7-linux-x86_64.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/python_hostlist-1.14-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/pymongo-2.7.2-py2.7-linux-x86_64.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/radical.utils-0.7.7-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/saga_python-0.18-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/GridDataFormats-0.2.4-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/apache_libcloud-0.15.1-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg',
'/work/n02/n02/eike/firedrake', '/work/n02/n02/eike/PyOP2',
'/work/n02/n02/eike/firedrake-bench', '/work/n02/n02/eike/pybench',
'/work/y07/y07/fdrake/firedrake/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/PyOP2/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/ufl/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/scientificpython/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/psutil/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/mpi4py/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/instant/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/fiat/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/ffc/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/petsc/arch-linux2-cxx-opt/lib/python2.7/site-packages',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7',
'/opt/cray/sdb/1.0-1.0501.48084.4.48.ari/lib64/py',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python27.zip',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/plat-linux2',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-tk',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-old',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-dynload',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/PIL',
'/opt/cray/sdb/1.0-1.0501.48084.4.48.ari/lib64/py']

On 15/09/14 09:37, Florian Rathgeber wrote:

On 15/09/14 09:20, Eike Mueller wrote:

Hi Florian,

I'm having another go at this now.

export
PYTHONPATH=${FDRAKE_DIR}/petsc4py/lib/python2.7/site-packages:${PYTHONPATH}
export
PYTHONPATH=${FDRAKE_DIR}/ufl/lib/python2.7/site-packages/:${PYTHONPATH}
export
PYTHONPATH=${FDRAKE_DIR}/fiat/lib/python2.7/site-packages/:${PYTHONPATH}
export
PYTHONPATH=${FDRAKE_DIR}/ffc/lib/python2.7/site-packages/:${PYTHONPATH}

I added this to the template file on the wiki.

This *should* be added by loading fdrake-python-env

* The https cloning issue I believe is resolved, at least I can't
reproduce it anymore.

All seems to work fine for me now.

* The PyOP2 installation issue related to NumPy is a bug [1] in
setuptools 2.2 which comes with anaconda. For now, the best thing to do
is to remove/comment line 138 in PyOP2/setup.py which declares the setup
dependency on NumPy.

I commented out the line  'setup_requires=setup_requires,' in setup.py
and the PyOP2 installation goes through without problems.

* The anaconda/environment modules conflict is resolved: I have removed
ANACONDA_LIB from the LD_LIBRARY_PATH and added it to LD_RUN_PATH
instead. This means Python extension modules get an RPATH to that
directory baked in and therefore can find the right libpython2.7.so
without any LD_LIBRARY_PATH being set (which is preferable anyway).

Works for me now, I can run the module command even after loading the
firedrake environment.

Unfortunately, Firedrake's setup.py manually sets an RPATH for PETSc and
therefore the LD_RUN_PATH is ignored. This patch is required:

diff --git a/setup.py b/setup.py
index 00838f8..350d51d 100644
--- a/setup.py
+++ b/setup.py
@@ -70,6 +70,7 @@ setup(name='firedrake',
                            include_dirs=include_dirs,
                            libraries=["petsc"],
                            extra_link_args=["-L%s/lib" % d for d in
petsc_dirs] +
-                             ["-Wl,-rpath,%s/lib" % d for d in
petsc_dirs]),
+                             ["-Wl,-rpath,%s/lib" % d for d in
petsc_dirs] +
+                             ["-Wl,-rpath,%s/lib" % sys.prefix]),
                  Extension('evtk.cevtk', evtk_sources,
                            include_dirs=[np.get_include()])])

Can you send me that patch as a separate text file, please? When I
copied it out of your email it didn't work.

I have pushed this to firedrake master.

Still something not quite right with the python. When I run a 'Hello
World!' test, i.e.

import sys
print 'Hello World!'
sys.exit(0)

I still get

python: error while loading shared libraries: libpython2.7.so.1.0:
cannot open shared object file: No such file or directory

That works for me. Not sure why this would fail since you're not loading
an extension module. Is this on the login node with anaconda?

but I guess this is due to the unresolved issue with anaconda that you
reported to the helpdesk, so I will wait for what comes back from them.

Indeed, as reported to helpdesk, it was premature thinking the
LD_LIBRARY_PATH was no longer required. I'm currently stuck on mpi4py
even though all the RPATHs are correctly set afaict. For now you'll need
to add $ANANCONDA_LIBS to LD_LIBRARY_PATH in your firedrake.tpl

Florian

Thanks,

Eike

For FFC it's more tricky, still trying to figure out what's needed.

Keep us posted on how things are progressing!

Florian

I will go ahead and edit the wiki.

Eike


On 8 Sep 2014, at 15:53, Eike Mueller <E.Mueller@bath.ac.uk
<mailto:E.Mueller@bath.ac.uk>
<mailto:E.Mueller@bath.ac.uk>> wrote:

Hi Florian,

thanks a lot for your reply and for updating the wiki, I got a bit
further now.

You're exactly right again: it seems that anaconda contains a
version of
the TCL libraries which are incompatible with those use by the
environment modules. I'm not sure how to work around this, since we
need
the anaconda libs (as you figured out later) - will open a ticket.

ok, I will use the environment script as it is and not worry about the
fact that the anaconda library breaks the module command at the
moment. Let's see if the support can figure something out.

I've changed the wiki page to clone via https, thanks!

It works with the updated instructions on the webpage. Strangely, if I
clone a repository into a directory, delete this directory and then
try to clone again with exactly the same command, nothing happens.

This seems to happen because (for reasons I can't figure out)
setuptools
doesn't detect that NumPy is already installed and builds it from
source. The primary cause of the error you're seeing is this:
https://stackoverflow.com/a/21621493

I will ignore this then, and just call the setup.py script twice in
the PyOP2 directory.

This is because you've removed the anaconda lib path, which also
contains the Python libraries themselves, from LD_LIBRARY_PATH

I tried a simpler hello-world script where I swap python -> anaconda
and run a python script. This does work.
Still trying to run the poisson example from the benchmark repository,
but the ARCHER queues seem to be terribly slow, even if I just run on
one node on the debug queue :-(

Absolutely, feel free to edit the wiki if you find more errors or
things
that are unclear.

I think I already have access to the repository, so will edit if
necessary.

Thanks a lot,

Eike

Florian

Cheers,

Eike

*** content of firedrake.env ***

module swap PrgEnv-cray PrgEnv-gnu
module unload python
module add anaconda

# Add fdrake module path
export FDRAKE_DIR=/work/y07/y07/fdrake
module use "$FDRAKE_DIR/modules"

# Load build environment
module load fdrake-build-env
module load fdrake-python-env

LD_LIBRARY_PATH=`echo $LD_LIBRARY_PATH | sed
's/\/work\/y07\/y07\/cse\/anaconda\/1.9.2\/lib//g'`

export
PYTHONPATH=$WORK/firedrake-bench:$WORK/firedrake-bench/pybench:$WORK/firedrake:$WORK/PyOP2:$PYTHONPATH

_______________________________________________
firedrake mailing list
firedrake@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/firedrake