Re: [firedrake] installing firedrake on a cluster

5 Apr 2017

      Actually, I'm back to step one.

With the last error message I wasn't on the right cluster.

When I go on the cluster that actually does computing I get that the numpy arrays are the wrong size.

Any advice of what I can try would be greatly appreciated.

Cheers, Francis
...
...
...
from firedrake import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/fpoulin/software/firedrake/src/firedrake/firedrake/__init__.py", line 26, in <module>
    from pyop2 import op2                                           # noqa
  File "/home/fpoulin/software/firedrake/src/PyOP2/pyop2/__init__.py", line 5, in <module>
    from pyop2.op2 import *  # noqa
  File "/home/fpoulin/software/firedrake/src/PyOP2/pyop2/op2.py", line 43, in <module>
    from pyop2.base import i                      # noqa: F401
  File "/home/fpoulin/software/firedrake/src/PyOP2/pyop2/base.py", line 57, in <module>
    from pyop2.sparsity import build_sparsity
  File "__init__.pxd", line 155, in init pyop2.sparsity (pyop2/sparsity.cpp:24410)
ValueError: numpy.dtype has the wrong size, try recompiling. Expected 88, got 96
------------------
Francis Poulin
Associate Professor
Department of Applied Mathematics
University of Waterloo

email:           fpoulin@uwaterloo.ca
Web:            https://uwaterloo.ca/poulin-research-group/
Telephone:  +1 519 888 4567 x32637

________________________________
From: firedrake-bounces@imperial.ac.uk [firedrake-bounces@imperial.ac.uk] on behalf of Francis Poulin [fpoulin@uwaterloo.ca]
Sent: Wednesday, April 05, 2017 9:20 AM
To: firedrake@imperial.ac.uk
Subject: Re: [firedrake] installing firedrake on a cluster

I decided to update firedrake on the cluster and then import firedrake to see if tha twas any better.

Now I seem to get getting a different error.

Anyone have a suggestion as to what I can try to do to fix this?

Cheers,
Francis
...
...
...
from firedrake import *

An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your MPI job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          orc-login2 (PID 24823)
  MPI_COMM_WORLD rank: 0

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/fpoulin/software/firedrake/src/firedrake/firedrake/__init__.py", line 31, in <module>
    from firedrake.checkpointing import *
  File "/home/fpoulin/software/firedrake/src/firedrake/firedrake/checkpointing.py", line 4, in <module>
    from firedrake import hdf5interface as h5i
  File "firedrake/hdf5interface.pyx", line 3, in init firedrake.hdf5interface (firedrake/hdf5interface.c:2333)
  File "/home/fpoulin/software/firedrake/src/h5py/h5py/__init__.py", line 28, in <module>
    raise ImportError("You cannot import h5py from inside the install directory.\nChange to another directory first.")
ImportError: You cannot import h5py from inside the install directory.
Change to another directory first.

------------------
Francis Poulin
Associate Professor
Department of Applied Mathematics
University of Waterloo

email:           fpoulin@uwaterloo.ca
Web:            https://uwaterloo.ca/poulin-research-group/
Telephone:  +1 519 888 4567 x32637

________________________________
From: Francis Poulin
Sent: Saturday, April 01, 2017 2:31 PM
To: firedrake@imperial.ac.uk
Subject: RE: [firedrake] installing firedrake on a cluster

Hello,

I installed firedrake on my cluster and that seemed to go ok in that I didn't get any error messages.

However, when I open the virtual environment, open python and try and import firedrake I get an error.   See below.

It seems unhappy with the size if numpy data types. I did an "pip install -U numpy" in case that was a problem but that didn't seem to help.

It is getting 96 and expecting 88.  Any ideas what the problem might be?

Cheers, Francis
...
...
...
from firedrake import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/fpoulin/software/firedrake/src/firedrake/firedrake/__init__.py", line 26, in <module>
    from pyop2 import op2                                           # noqa
  File "/home/fpoulin/software/firedrake/src/PyOP2/pyop2/__init__.py", line 5, in <module>
    from pyop2.op2 import *  # noqa
  File "/home/fpoulin/software/firedrake/src/PyOP2/pyop2/op2.py", line 43, in <module>
    from pyop2.base import i                      # noqa: F401
  File "/home/fpoulin/software/firedrake/src/PyOP2/pyop2/base.py", line 57, in <module>
    from pyop2.sparsity import build_sparsity
  File "__init__.pxd", line 155, in init pyop2.sparsity (pyop2/sparsity.cpp:24410)
ValueError: numpy.dtype has the wrong size, try recompiling. Expected 88, got 96
------------------
Francis Poulin
Associate Professor
Department of Applied Mathematics
University of Waterloo

email:           fpoulin@uwaterloo.ca
Web:            https://uwaterloo.ca/poulin-research-group/
Telephone:  +1 519 888 4567 x32637

________________________________
From: firedrake-bounces@imperial.ac.uk [firedrake-bounces@imperial.ac.uk] on behalf of David Ham [David.Ham@imperial.ac.uk]
Sent: Friday, March 31, 2017 8:22 AM
To: firedrake
Subject: Re: [firedrake] installing firedrake on a cluster

Hi Francis,

This is not the entire log. You've only pasted the Python exception. That Python exception was caused by something failing in PETSc's build system. However we can't see from here what that is. Please attach the whole log.

Many thanks,

David

On Fri, 31 Mar 2017 at 13:09 Francis Poulin <fpoulin@uwaterloo.ca<mailto:fpoulin@uwaterloo.ca>> wrote:
Thanks again.  That was very helpful.

It seemed to make it to the petsc install before it gave up.  I highlighted in read what appeared to me in red.

Any ideas what went wrong and how I could fix it?

Cheers, Francis

   Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/local/tmp/pip-GY2H_M-build/setup.py", line 302, in <module>
        **metadata)
      File "/opt/sharcnet/python/2.7.8/intel/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/opt/sharcnet/python/2.7.8/intel/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/opt/sharcnet/python/2.7.8/intel/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/local/tmp/pip-GY2H_M-build/setup.py", line 218, in run
        config(prefix, self.dry_run)
      File "/local/tmp/pip-GY2H_M-build/setup.py", line 148, in config
        if status != 0: raise RuntimeError(status)
    RuntimeError: 256

    ----------------------------------------
Command "/home/fpoulin/software/firedrake/bin/python -u -c "import setuptools, tokenize;__file__='/local/tmp/pip-GY2H_M-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-fXQE7k-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/fpoulin/software/firedrake/include/site/python2.7/petsc" failed with error code 1 in /local/tmp/pip-GY2H_M-build/
Traceback (most recent call last):
  File "firedrake-install", line 1234, in <module>
    install("petsc/")
  File "firedrake-install", line 678, in install
    run_pip_install(["--ignore-installed", package])
  File "firedrake-install", line 548, in run_pip_install
    check_call(pipinstall + pipargs)

  File "firedrake-install", line 369, in check_call
    subprocess.check_call(arguments, env=env)
  File "/opt/sharcnet/python/2.7.8/intel/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/fpoulin/software/firedrake/bin/pip', 'install', '--no-binary', 'mpi4py', '--no-deps', '--ignore-installed', 'petsc/']' returned non-zero exit status 1

------------------
Francis Poulin
Associate Professor
Department of Applied Mathematics
University of Waterloo

email:           fpoulin@uwaterloo.ca<mailto:fpoulin@uwaterloo.ca>
Web:            https://uwaterloo.ca/poulin-research-group/
Telephone:  +1 519 888 4567 x32637<tel:(519)%20888-4567>

________________________________
From: firedrake-bounces@imperial.ac.uk<mailto:firedrake-bounces@imperial.ac.uk> [firedrake-bounces@imperial.ac.uk<mailto:firedrake-bounces@imperial.ac.uk>] on behalf of David Ham [David.Ham@imperial.ac.uk<mailto:David.Ham@imperial.ac.uk>]
Sent: Friday, March 31, 2017 5:55 AM
To: firedrake

Subject: Re: [firedrake] installing firedrake on a cluster
Hi Francis and thanks Justin for the good advice!

First, you almost certainly want to:

unset PYTHONPATH

instead of telling Firedrake to honour it. This is because it's unlikely that there is something in your PYTHONPATH which you want the virtualenv to see (this is why having PYTHONPATH set triggers this error).

Next, assuming that you managed to install pip using Justin's instructions (or by having the sysadmin do it), you can try running:

pip install --user virtualenv

yourself. If that works, try running the install script again. If it fails, then show us the error message and we might be able to understand what's gone wrong.

Regards,

David

On Fri, 31 Mar 2017 at 03:43 Justin Chang <jychang48@gmail.com<mailto:jychang48@gmail.com>> wrote:
Francis, I assume you tried get-pip.py<https://bootstrap.pypa.io/get-pip.py>?

If you simply run the above with system python (e.g., 'python get-pip.py') it will return an error because it is trying to install pip into the root directory which I am assuming you have no access to. What you need to do is install pip locally. I did this on my local cluster through 'python get-pip.py --user' which will install pip into $HOME/.local. I can't remember what I did next but I think I had to ensure that this .local directory was visible through PATH:

export PATH=$HOME/.local:$HOME/.local/bin:$PATH

and then pip worked.

Justin

On Thu, Mar 30, 2017 at 9:32 PM, Francis Poulin <fpoulin@uwaterloo.ca<mailto:fpoulin@uwaterloo.ca>> wrote:
Thanks again.

It would appear that I don't have pip installed.   (see below)

I found instructions on their site on how to do that and of course it doesn't work for some reason.  I will get to to him me set up pip and then try again tomorrow and hope I can get a bit further.  I will let you know things progress, when they progress that is.

Cheers, Francis

[fpoulin@orc129 software]$ apt-get --version
-bash: apt-get: command not found
[fpoulin@orc129 software]$ python firedrake-install --honour-pythonpath --no_package_manager
apt-get not found or disabled. Proceeding on the rash assumption that your compiled dependencies are in place.
If this is not the case, please install the following and try again:
* A C and C++ compiler (for example gcc/g++ or clang), GNU make
* A Fortran compiler (for PETSc)
* MPI
* Blas and Lapack
* Git, Mercurial
* Python version 2.7
* pip and the Python headers
* autoconf, automake, libtool
* CMake
* zlib
Installing virtualenv.
Collecting virtualenv
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', error(101, 'Network is unreachable'))': /simple/virtualenv/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', error(101, 'Network is unreachable'))': /simple/virtualenv/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', error(101, 'Network is unreachable'))': /simple/virtualenv/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', error(101, 'Network is unreachable'))': /simple/virtualenv/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', error(101, 'Network is unreachable'))': /simple/virtualenv/
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', error(101, 'Network is unreachable'))': /simple/virtualenv/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', error(101, 'Network is unreachable'))': /simple/virtualenv/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', error(101, 'Network is unreachable'))': /simple/virtualenv/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', error(101, 'Network is unreachable'))': /simple/virtualenv/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', error(101, 'Network is unreachable'))': /simple/virtualenv/
  Could not find a version that satisfies the requirement virtualenv (from versions: )
No matching distribution found for virtualenv
Traceback (most recent call last):
  File "firedrake-install", line 1172, in <module>
    check_call(["pip", "install", "--user", "virtualenv"])
  File "firedrake-install", line 369, in check_call
    subprocess.check_call(arguments, env=env)
  File "/opt/sharcnet/python/2.7.8/intel/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip', 'install', '--user', 'virtualenv']' returned non-zero exit status 1

_______________________________________________
firedrake mailing list
firedrake@imperial.ac.uk<mailto:firedrake@imperial.ac.uk>
https://mailman.ic.ac.uk/mailman/listinfo/firedrake

--
Dr David Ham
Department of Mathematics
Imperial College London
--
Dr David Ham
Department of Mathematics
Imperial College London