Hello, I am trying to implement a Dirac backend for the software we use at T2K to access the grid data. This software is written in Python and now I am facing the problem that Dirac comes with its own Python distribution that does *not* play nicely with either the system Python or virtual environments. So my question is this: Is there a(n easy) way to make Dirac compatible with other Python software? Maybe a way to install the Dirac client in a regular virtual Python environment? Cheers Lukas -- Lukas KOCH Particle Physics Department STFC Rutherford Appleton Laboratory Mail: lukas.koch@stfc.ac.uk Tel: +44 (0) 1235 44 x6265
Hi Lukas, The easiest way of doing this is to treat the DIRAC python environment as if it's a virtualenv and install whatever you need in to that (rather than the other way around). # Install DIRAC UI & then start with a clean env... # Get a subshell bash source bashrc unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip pip install -e t2kdm # or whatever exit source bashrc # You should now have a python environment with your modules & DIRAC Regards, Simon On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote:
Hello,
I am trying to implement a Dirac backend for the software we use at T2K to access the grid data. This software is written in Python and now I am facing the problem that Dirac comes with its own Python distribution that does *not* play nicely with either the system Python or virtual environments.
So my question is this: Is there a(n easy) way to make Dirac compatible with other Python software? Maybe a way to install the Dirac client in a regular virtual Python environment?
Cheers Lukas
Hello Simon, I tried to do as you said, but pip does not seem to work after sourcing DIRAC's bashrc: $ pip Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", line 7, in <module> from pip import main ImportError: cannot import name main Cheers Lukas Am 09/11/2018 um 15:16 schrieb Simon Fayer:
Hi Lukas,
The easiest way of doing this is to treat the DIRAC python environment as if it's a virtualenv and install whatever you need in to that (rather than the other way around).
# Install DIRAC UI & then start with a clean env... # Get a subshell bash source bashrc unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip pip install -e t2kdm # or whatever exit source bashrc # You should now have a python environment with your modules & DIRAC
Regards, Simon
On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote:
Hello,
I am trying to implement a Dirac backend for the software we use at T2K to access the grid data. This software is written in Python and now I am facing the problem that Dirac comes with its own Python distribution that does *not* play nicely with either the system Python or virtual environments.
So my question is this: Is there a(n easy) way to make Dirac compatible with other Python software? Maybe a way to install the Dirac client in a regular virtual Python environment?
Cheers Lukas
-- Lukas KOCH Particle Physics Department STFC Rutherford Appleton Laboratory Mail: lukas.koch@stfc.ac.uk Tel: +44 (0) 1235 44 x6265
Hi Lukas, Did you install DIRAC recently? Do you have any other python stuff installed in your user directory that may be interfering (--user installed pip or PYTHONPATH in ~/.bashrc)? I tried it from a clean install (on CentOS6) and it seems to work for me... $ mkdir testdir; cd testdir $ wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/d... $ chmod u+x dirac-install $ ./dirac-install -r v6r20p15 -i 27 -g v14r1 $ source bashrc $ unset REQUESTS_CA_BUNDLE SSL_CERT_DIR $ pip help Usage: pip <command> [options] ... Regards, Simon On Fri, Nov 09, 2018 at 03:46:19PM +0000, Lukas Koch wrote:
Hello Simon,
I tried to do as you said, but pip does not seem to work after sourcing DIRAC's bashrc:
$ pip Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", line 7, in <module> from pip import main ImportError: cannot import name main
Cheers Lukas
Am 09/11/2018 um 15:16 schrieb Simon Fayer:
Hi Lukas,
The easiest way of doing this is to treat the DIRAC python environment as if it's a virtualenv and install whatever you need in to that (rather than the other way around).
# Install DIRAC UI & then start with a clean env... # Get a subshell bash source bashrc unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip pip install -e t2kdm # or whatever exit source bashrc # You should now have a python environment with your modules & DIRAC
Regards, Simon
On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote:
Hello,
I am trying to implement a Dirac backend for the software we use at T2K to access the grid data. This software is written in Python and now I am facing the problem that Dirac comes with its own Python distribution that does *not* play nicely with either the system Python or virtual environments.
So my question is this: Is there a(n easy) way to make Dirac compatible with other Python software? Maybe a way to install the Dirac client in a regular virtual Python environment?
Cheers Lukas
Hello Simon, I did install it today with exactly the same command line as you quoted. I do have locally installed python packages, but I assume those will not affect the DIRAC Python. At least they should not, if I am to treat it like a virtual environment. Anyway it seems like it is trying to use the right pip: $ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip And the PYTHONPATH looks reasonable as well: $ env | grep PYTHON PYTHONUNBUFFERED=yes PYTHONOPTIMIZE=x PYTHONPATH=/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC: This is on a machine running SL6, by the way. Maybe that has something to do with it. Cheers Lukas Am 09/11/2018 um 16:12 schrieb Simon Fayer:
Hi Lukas,
Did you install DIRAC recently? Do you have any other python stuff installed in your user directory that may be interfering (--user installed pip or PYTHONPATH in ~/.bashrc)?
I tried it from a clean install (on CentOS6) and it seems to work for me...
$ mkdir testdir; cd testdir $ wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/d... $ chmod u+x dirac-install $ ./dirac-install -r v6r20p15 -i 27 -g v14r1 $ source bashrc $ unset REQUESTS_CA_BUNDLE SSL_CERT_DIR $ pip help
Usage: pip <command> [options]
...
Regards, Simon
On Fri, Nov 09, 2018 at 03:46:19PM +0000, Lukas Koch wrote:
Hello Simon,
I tried to do as you said, but pip does not seem to work after sourcing DIRAC's bashrc:
$ pip Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", line 7, in <module> from pip import main ImportError: cannot import name main
Cheers Lukas
Am 09/11/2018 um 15:16 schrieb Simon Fayer:
Hi Lukas,
The easiest way of doing this is to treat the DIRAC python environment as if it's a virtualenv and install whatever you need in to that (rather than the other way around).
# Install DIRAC UI & then start with a clean env... # Get a subshell bash source bashrc unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip pip install -e t2kdm # or whatever exit source bashrc # You should now have a python environment with your modules & DIRAC
Regards, Simon
On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote:
Hello,
I am trying to implement a Dirac backend for the software we use at T2K to access the grid data. This software is written in Python and now I am facing the problem that Dirac comes with its own Python distribution that does *not* play nicely with either the system Python or virtual environments.
So my question is this: Is there a(n easy) way to make Dirac compatible with other Python software? Maybe a way to install the Dirac client in a regular virtual Python environment?
Cheers Lukas
-- Lukas KOCH Particle Physics Department STFC Rutherford Appleton Laboratory Mail: lukas.koch@stfc.ac.uk Tel: +44 (0) 1235 44 x6265
Hi Lukas, I don't think SL6 should be substantially different to CentOS6. You don't happen to have a directory called "pip" in your CWD do you? You could try running the following to see where it's finding the pip module from (i.e. to check it's using the one from the DIRAC bundle): $ python -v -c "import pip" 2>&1 | grep pip | head -n 3 Regards, Simon On Fri, Nov 09, 2018 at 04:48:11PM +0000, Lukas Koch wrote:
Hello Simon,
I did install it today with exactly the same command line as you quoted. I do have locally installed python packages, but I assume those will not affect the DIRAC Python. At least they should not, if I am to treat it like a virtual environment. Anyway it seems like it is trying to use the right pip:
$ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
And the PYTHONPATH looks reasonable as well:
$ env | grep PYTHON PYTHONUNBUFFERED=yes PYTHONOPTIMIZE=x PYTHONPATH=/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC:
This is on a machine running SL6, by the way. Maybe that has something to do with it.
Cheers Lukas
Am 09/11/2018 um 16:12 schrieb Simon Fayer:
Hi Lukas,
Did you install DIRAC recently? Do you have any other python stuff installed in your user directory that may be interfering (--user installed pip or PYTHONPATH in ~/.bashrc)?
I tried it from a clean install (on CentOS6) and it seems to work for me...
$ mkdir testdir; cd testdir $ wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/d... $ chmod u+x dirac-install $ ./dirac-install -r v6r20p15 -i 27 -g v14r1 $ source bashrc $ unset REQUESTS_CA_BUNDLE SSL_CERT_DIR $ pip help
Usage: pip <command> [options]
...
Regards, Simon
On Fri, Nov 09, 2018 at 03:46:19PM +0000, Lukas Koch wrote:
Hello Simon,
I tried to do as you said, but pip does not seem to work after sourcing DIRAC's bashrc:
$ pip Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", line 7, in <module> from pip import main ImportError: cannot import name main
Cheers Lukas
Am 09/11/2018 um 15:16 schrieb Simon Fayer:
Hi Lukas,
The easiest way of doing this is to treat the DIRAC python environment as if it's a virtualenv and install whatever you need in to that (rather than the other way around).
# Install DIRAC UI & then start with a clean env... # Get a subshell bash source bashrc unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip pip install -e t2kdm # or whatever exit source bashrc # You should now have a python environment with your modules & DIRAC
Regards, Simon
On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote:
Hello,
I am trying to implement a Dirac backend for the software we use at T2K to access the grid data. This software is written in Python and now I am facing the problem that Dirac comes with its own Python distribution that does *not* play nicely with either the system Python or virtual environments.
So my question is this: Is there a(n easy) way to make Dirac compatible with other Python software? Maybe a way to install the Dirac client in a regular virtual Python environment?
Cheers Lukas
Ok, so it seems like it is trying to import the pip module from my ~/.local folder: $ python -v -c "import pip" 2>&1 | grep pip | head -n 3 import pip # directory /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip # /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo matches /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.py import pip # precompiled from /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo Which is weird, since that path is not in my PYTHONPATH and both python and pip seem to come from the right place: $ which python /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/python $ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip I guess it must be one of the default paths used by Python. It looks like it comes just before the DIRAC folders priority though: $ python -c 'import sys; print sys.path' ['', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python27.zip', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/plat-linux2', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-tk', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-old', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-dynload', '/home/ppd/kdf77245/.local/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages/GSI-0.6.5-py2.7-linux-x86_64.egg'] Am 09/11/2018 um 16:55 schrieb Simon Fayer:
Hi Lukas,
I don't think SL6 should be substantially different to CentOS6. You don't happen to have a directory called "pip" in your CWD do you?
You could try running the following to see where it's finding the pip module from (i.e. to check it's using the one from the DIRAC bundle):
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3
Regards, Simon
On Fri, Nov 09, 2018 at 04:48:11PM +0000, Lukas Koch wrote:
Hello Simon,
I did install it today with exactly the same command line as you quoted. I do have locally installed python packages, but I assume those will not affect the DIRAC Python. At least they should not, if I am to treat it like a virtual environment. Anyway it seems like it is trying to use the right pip:
$ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
And the PYTHONPATH looks reasonable as well:
$ env | grep PYTHON PYTHONUNBUFFERED=yes PYTHONOPTIMIZE=x PYTHONPATH=/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC:
This is on a machine running SL6, by the way. Maybe that has something to do with it.
Cheers Lukas
Am 09/11/2018 um 16:12 schrieb Simon Fayer:
Hi Lukas,
Did you install DIRAC recently? Do you have any other python stuff installed in your user directory that may be interfering (--user installed pip or PYTHONPATH in ~/.bashrc)?
I tried it from a clean install (on CentOS6) and it seems to work for me...
$ mkdir testdir; cd testdir $ wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/d... $ chmod u+x dirac-install $ ./dirac-install -r v6r20p15 -i 27 -g v14r1 $ source bashrc $ unset REQUESTS_CA_BUNDLE SSL_CERT_DIR $ pip help
Usage: pip <command> [options]
...
Regards, Simon
On Fri, Nov 09, 2018 at 03:46:19PM +0000, Lukas Koch wrote:
Hello Simon,
I tried to do as you said, but pip does not seem to work after sourcing DIRAC's bashrc:
$ pip Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", line 7, in <module> from pip import main ImportError: cannot import name main
Cheers Lukas
Am 09/11/2018 um 15:16 schrieb Simon Fayer:
Hi Lukas,
The easiest way of doing this is to treat the DIRAC python environment as if it's a virtualenv and install whatever you need in to that (rather than the other way around).
# Install DIRAC UI & then start with a clean env... # Get a subshell bash source bashrc unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip pip install -e t2kdm # or whatever exit source bashrc # You should now have a python environment with your modules & DIRAC
Regards, Simon
On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote:
Hello,
I am trying to implement a Dirac backend for the software we use at T2K to access the grid data. This software is written in Python and now I am facing the problem that Dirac comes with its own Python distribution that does *not* play nicely with either the system Python or virtual environments.
So my question is this: Is there a(n easy) way to make Dirac compatible with other Python software? Maybe a way to install the Dirac client in a regular virtual Python environment?
Cheers Lukas
-- Lukas KOCH Particle Physics Department STFC Rutherford Appleton Laboratory Mail: lukas.koch@stfc.ac.uk Tel: +44 (0) 1235 44 x6265
You could try disabling using the ~/.local folder by doing: export PYTHONNOUSERSITE=1 That should make its behaviour more like a real virtualenv... Regards, Simon On Fri, Nov 09, 2018 at 05:15:13PM +0000, Lukas Koch wrote:
Ok, so it seems like it is trying to import the pip module from my ~/.local folder:
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3 import pip # directory /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip # /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo matches /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.py import pip # precompiled from /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo
Which is weird, since that path is not in my PYTHONPATH and both python and pip seem to come from the right place:
$ which python /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/python $ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
I guess it must be one of the default paths used by Python. It looks like it comes just before the DIRAC folders priority though:
$ python -c 'import sys; print sys.path' ['', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python27.zip', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/plat-linux2', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-tk', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-old', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-dynload', '/home/ppd/kdf77245/.local/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages/GSI-0.6.5-py2.7-linux-x86_64.egg']
Am 09/11/2018 um 16:55 schrieb Simon Fayer:
Hi Lukas,
I don't think SL6 should be substantially different to CentOS6. You don't happen to have a directory called "pip" in your CWD do you?
You could try running the following to see where it's finding the pip module from (i.e. to check it's using the one from the DIRAC bundle):
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3
Regards, Simon
On Fri, Nov 09, 2018 at 04:48:11PM +0000, Lukas Koch wrote:
Hello Simon,
I did install it today with exactly the same command line as you quoted. I do have locally installed python packages, but I assume those will not affect the DIRAC Python. At least they should not, if I am to treat it like a virtual environment. Anyway it seems like it is trying to use the right pip:
$ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
And the PYTHONPATH looks reasonable as well:
$ env | grep PYTHON PYTHONUNBUFFERED=yes PYTHONOPTIMIZE=x PYTHONPATH=/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC:
This is on a machine running SL6, by the way. Maybe that has something to do with it.
Cheers Lukas
Am 09/11/2018 um 16:12 schrieb Simon Fayer:
Hi Lukas,
Did you install DIRAC recently? Do you have any other python stuff installed in your user directory that may be interfering (--user installed pip or PYTHONPATH in ~/.bashrc)?
I tried it from a clean install (on CentOS6) and it seems to work for me...
$ mkdir testdir; cd testdir $ wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/d... $ chmod u+x dirac-install $ ./dirac-install -r v6r20p15 -i 27 -g v14r1 $ source bashrc $ unset REQUESTS_CA_BUNDLE SSL_CERT_DIR $ pip help
Usage: pip <command> [options]
...
Regards, Simon
On Fri, Nov 09, 2018 at 03:46:19PM +0000, Lukas Koch wrote:
Hello Simon,
I tried to do as you said, but pip does not seem to work after sourcing DIRAC's bashrc:
$ pip Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", line 7, in <module> from pip import main ImportError: cannot import name main
Cheers Lukas
Am 09/11/2018 um 15:16 schrieb Simon Fayer:
Hi Lukas,
The easiest way of doing this is to treat the DIRAC python environment as if it's a virtualenv and install whatever you need in to that (rather than the other way around).
# Install DIRAC UI & then start with a clean env... # Get a subshell bash source bashrc unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip pip install -e t2kdm # or whatever exit source bashrc # You should now have a python environment with your modules & DIRAC
Regards, Simon
On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote: > Hello, > > I am trying to implement a Dirac backend for the software we use at T2K > to access the grid data. This software is written in Python and now I am > facing the problem that Dirac comes with its own Python distribution > that does *not* play nicely with either the system Python or virtual > environments. > > So my question is this: Is there a(n easy) way to make Dirac compatible > with other Python software? Maybe a way to install the Dirac client in a > regular virtual Python environment? > > Cheers > Lukas >
Hi Simon, your solution is much cleaner than mine, thanks! I appended these lines to the stock bashrc and it seems to work for now: # Turn DIRAC into something resembling a virtualenv unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip export PYTHONNOUSERSITE=1 Cheers Lukas Am 09/11/2018 um 17:20 schrieb Simon Fayer:
You could try disabling using the ~/.local folder by doing:
export PYTHONNOUSERSITE=1
That should make its behaviour more like a real virtualenv...
Regards, Simon
On Fri, Nov 09, 2018 at 05:15:13PM +0000, Lukas Koch wrote:
Ok, so it seems like it is trying to import the pip module from my ~/.local folder:
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3 import pip # directory /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip # /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo matches /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.py import pip # precompiled from /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo
Which is weird, since that path is not in my PYTHONPATH and both python and pip seem to come from the right place:
$ which python /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/python $ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
I guess it must be one of the default paths used by Python. It looks like it comes just before the DIRAC folders priority though:
$ python -c 'import sys; print sys.path' ['', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python27.zip', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/plat-linux2', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-tk', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-old', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-dynload', '/home/ppd/kdf77245/.local/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages/GSI-0.6.5-py2.7-linux-x86_64.egg']
Am 09/11/2018 um 16:55 schrieb Simon Fayer:
Hi Lukas,
I don't think SL6 should be substantially different to CentOS6. You don't happen to have a directory called "pip" in your CWD do you?
You could try running the following to see where it's finding the pip module from (i.e. to check it's using the one from the DIRAC bundle):
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3
Regards, Simon
On Fri, Nov 09, 2018 at 04:48:11PM +0000, Lukas Koch wrote:
Hello Simon,
I did install it today with exactly the same command line as you quoted. I do have locally installed python packages, but I assume those will not affect the DIRAC Python. At least they should not, if I am to treat it like a virtual environment. Anyway it seems like it is trying to use the right pip:
$ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
And the PYTHONPATH looks reasonable as well:
$ env | grep PYTHON PYTHONUNBUFFERED=yes PYTHONOPTIMIZE=x PYTHONPATH=/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC:
This is on a machine running SL6, by the way. Maybe that has something to do with it.
Cheers Lukas
Am 09/11/2018 um 16:12 schrieb Simon Fayer:
Hi Lukas,
Did you install DIRAC recently? Do you have any other python stuff installed in your user directory that may be interfering (--user installed pip or PYTHONPATH in ~/.bashrc)?
I tried it from a clean install (on CentOS6) and it seems to work for me...
$ mkdir testdir; cd testdir $ wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/d... $ chmod u+x dirac-install $ ./dirac-install -r v6r20p15 -i 27 -g v14r1 $ source bashrc $ unset REQUESTS_CA_BUNDLE SSL_CERT_DIR $ pip help
Usage: pip <command> [options]
...
Regards, Simon
On Fri, Nov 09, 2018 at 03:46:19PM +0000, Lukas Koch wrote:
Hello Simon,
I tried to do as you said, but pip does not seem to work after sourcing DIRAC's bashrc:
$ pip Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", line 7, in <module> from pip import main ImportError: cannot import name main
Cheers Lukas
Am 09/11/2018 um 15:16 schrieb Simon Fayer: > Hi Lukas, > > The easiest way of doing this is to treat the DIRAC python environment as > if it's a virtualenv and install whatever you need in to that (rather than > the other way around). > > # Install DIRAC UI & then start with a clean env... > # Get a subshell > bash > source bashrc > unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip > pip install -e t2kdm # or whatever > exit > source bashrc > # You should now have a python environment with your modules & DIRAC > > Regards, > Simon > > > On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote: >> Hello, >> >> I am trying to implement a Dirac backend for the software we use at T2K >> to access the grid data. This software is written in Python and now I am >> facing the problem that Dirac comes with its own Python distribution >> that does *not* play nicely with either the system Python or virtual >> environments. >> >> So my question is this: Is there a(n easy) way to make Dirac compatible >> with other Python software? Maybe a way to install the Dirac client in a >> regular virtual Python environment? >> >> Cheers >> Lukas >>
-- Lukas KOCH Particle Physics Department STFC Rutherford Appleton Laboratory Mail: lukas.koch@stfc.ac.uk Tel: +44 (0) 1235 44 x6265
Possibly related question: After sourcing DIRAC's bashrc, the gfal-* commands no longer work: $ gfal-ls Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/gfal-ls", line 24, in <module> from gfal2_util.shell import Gfal2Shell ImportError: No module named gfal2_util.shell This is with both the original and my modified bashrc. Is that expected behaviour? Seems a bit strange since this is apparently the gfal command distributed with DIRAC. Cheers Lukas Am 09/11/2018 um 17:27 schrieb Lukas Koch:
Hi Simon,
your solution is much cleaner than mine, thanks! I appended these lines to the stock bashrc and it seems to work for now:
# Turn DIRAC into something resembling a virtualenv unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip export PYTHONNOUSERSITE=1
Cheers Lukas
Am 09/11/2018 um 17:20 schrieb Simon Fayer:
You could try disabling using the ~/.local folder by doing:
export PYTHONNOUSERSITE=1
That should make its behaviour more like a real virtualenv...
Regards, Simon
On Fri, Nov 09, 2018 at 05:15:13PM +0000, Lukas Koch wrote:
Ok, so it seems like it is trying to import the pip module from my ~/.local folder:
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3 import pip # directory /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip # /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo matches /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.py import pip # precompiled from /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo
Which is weird, since that path is not in my PYTHONPATH and both python and pip seem to come from the right place:
$ which python /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/python $ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
I guess it must be one of the default paths used by Python. It looks like it comes just before the DIRAC folders priority though:
$ python -c 'import sys; print sys.path' ['', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python27.zip', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/plat-linux2', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-tk', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-old', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-dynload', '/home/ppd/kdf77245/.local/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages/GSI-0.6.5-py2.7-linux-x86_64.egg']
Am 09/11/2018 um 16:55 schrieb Simon Fayer:
Hi Lukas,
I don't think SL6 should be substantially different to CentOS6. You don't happen to have a directory called "pip" in your CWD do you?
You could try running the following to see where it's finding the pip module from (i.e. to check it's using the one from the DIRAC bundle):
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3
Regards, Simon
On Fri, Nov 09, 2018 at 04:48:11PM +0000, Lukas Koch wrote:
Hello Simon,
I did install it today with exactly the same command line as you quoted. I do have locally installed python packages, but I assume those will not affect the DIRAC Python. At least they should not, if I am to treat it like a virtual environment. Anyway it seems like it is trying to use the right pip:
$ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
And the PYTHONPATH looks reasonable as well:
$ env | grep PYTHON PYTHONUNBUFFERED=yes PYTHONOPTIMIZE=x PYTHONPATH=/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC:
This is on a machine running SL6, by the way. Maybe that has something to do with it.
Cheers Lukas
Am 09/11/2018 um 16:12 schrieb Simon Fayer:
Hi Lukas,
Did you install DIRAC recently? Do you have any other python stuff installed in your user directory that may be interfering (--user installed pip or PYTHONPATH in ~/.bashrc)?
I tried it from a clean install (on CentOS6) and it seems to work for me...
$ mkdir testdir; cd testdir $ wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/d... $ chmod u+x dirac-install $ ./dirac-install -r v6r20p15 -i 27 -g v14r1 $ source bashrc $ unset REQUESTS_CA_BUNDLE SSL_CERT_DIR $ pip help
Usage: pip <command> [options]
...
Regards, Simon
On Fri, Nov 09, 2018 at 03:46:19PM +0000, Lukas Koch wrote: > Hello Simon, > > I tried to do as you said, but pip does not seem to work after sourcing > DIRAC's bashrc: > > $ pip > Traceback (most recent call last): > File > "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", > line 7, in <module> > from pip import main > ImportError: cannot import name main > > Cheers > Lukas > > Am 09/11/2018 um 15:16 schrieb Simon Fayer: >> Hi Lukas, >> >> The easiest way of doing this is to treat the DIRAC python environment as >> if it's a virtualenv and install whatever you need in to that (rather than >> the other way around). >> >> # Install DIRAC UI & then start with a clean env... >> # Get a subshell >> bash >> source bashrc >> unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip >> pip install -e t2kdm # or whatever >> exit >> source bashrc >> # You should now have a python environment with your modules & DIRAC >> >> Regards, >> Simon >> >> >> On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote: >>> Hello, >>> >>> I am trying to implement a Dirac backend for the software we use at T2K >>> to access the grid data. This software is written in Python and now I am >>> facing the problem that Dirac comes with its own Python distribution >>> that does *not* play nicely with either the system Python or virtual >>> environments. >>> >>> So my question is this: Is there a(n easy) way to make Dirac compatible >>> with other Python software? Maybe a way to install the Dirac client in a >>> regular virtual Python environment? >>> >>> Cheers >>> Lukas >>> >
-- Lukas KOCH Particle Physics Department STFC Rutherford Appleton Laboratory Mail: lukas.koch@stfc.ac.uk Tel: +44 (0) 1235 44 x6265
Hi Lukas, This is a bug in the bundling of the DIRAC externals: We've reported it back to the developers but it evidently hasn't been fixed yet. (The problem is that the gfal2 shell package is being installed in the python2.6 directory even if it's built for 2.7. A workaround is to add the misplaced directory to the python path: export PYTHONPATH=${PYTHONPATH}:${DIRAC}/Linux_x86_64_glibc-2.12/lib/python2.6/site-packages Regards, Simon On Fri, Nov 09, 2018 at 05:37:24PM +0000, Lukas Koch wrote:
Possibly related question:
After sourcing DIRAC's bashrc, the gfal-* commands no longer work:
$ gfal-ls Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/gfal-ls", line 24, in <module> from gfal2_util.shell import Gfal2Shell ImportError: No module named gfal2_util.shell
This is with both the original and my modified bashrc. Is that expected behaviour? Seems a bit strange since this is apparently the gfal command distributed with DIRAC.
Cheers Lukas
Am 09/11/2018 um 17:27 schrieb Lukas Koch:
Hi Simon,
your solution is much cleaner than mine, thanks! I appended these lines to the stock bashrc and it seems to work for now:
# Turn DIRAC into something resembling a virtualenv unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip export PYTHONNOUSERSITE=1
Cheers Lukas
Am 09/11/2018 um 17:20 schrieb Simon Fayer:
You could try disabling using the ~/.local folder by doing:
export PYTHONNOUSERSITE=1
That should make its behaviour more like a real virtualenv...
Regards, Simon
On Fri, Nov 09, 2018 at 05:15:13PM +0000, Lukas Koch wrote:
Ok, so it seems like it is trying to import the pip module from my ~/.local folder:
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3 import pip # directory /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip # /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo matches /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.py import pip # precompiled from /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo
Which is weird, since that path is not in my PYTHONPATH and both python and pip seem to come from the right place:
$ which python /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/python $ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
I guess it must be one of the default paths used by Python. It looks like it comes just before the DIRAC folders priority though:
$ python -c 'import sys; print sys.path' ['', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python27.zip', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/plat-linux2', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-tk', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-old', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-dynload', '/home/ppd/kdf77245/.local/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages/GSI-0.6.5-py2.7-linux-x86_64.egg']
Am 09/11/2018 um 16:55 schrieb Simon Fayer:
Hi Lukas,
I don't think SL6 should be substantially different to CentOS6. You don't happen to have a directory called "pip" in your CWD do you?
You could try running the following to see where it's finding the pip module from (i.e. to check it's using the one from the DIRAC bundle):
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3
Regards, Simon
On Fri, Nov 09, 2018 at 04:48:11PM +0000, Lukas Koch wrote:
Hello Simon,
I did install it today with exactly the same command line as you quoted. I do have locally installed python packages, but I assume those will not affect the DIRAC Python. At least they should not, if I am to treat it like a virtual environment. Anyway it seems like it is trying to use the right pip:
$ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
And the PYTHONPATH looks reasonable as well:
$ env | grep PYTHON PYTHONUNBUFFERED=yes PYTHONOPTIMIZE=x PYTHONPATH=/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC:
This is on a machine running SL6, by the way. Maybe that has something to do with it.
Cheers Lukas
Am 09/11/2018 um 16:12 schrieb Simon Fayer: > Hi Lukas, > > Did you install DIRAC recently? Do you have any other python stuff > installed in your user directory that may be interfering (--user installed > pip or PYTHONPATH in ~/.bashrc)? > > I tried it from a clean install (on CentOS6) and it seems to work for me... > > $ mkdir testdir; cd testdir > $ wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/d... > $ chmod u+x dirac-install > $ ./dirac-install -r v6r20p15 -i 27 -g v14r1 > $ source bashrc > $ unset REQUESTS_CA_BUNDLE SSL_CERT_DIR > $ pip help > > Usage: > pip <command> [options] > > ... > > Regards, > Simon > > > On Fri, Nov 09, 2018 at 03:46:19PM +0000, Lukas Koch wrote: >> Hello Simon, >> >> I tried to do as you said, but pip does not seem to work after sourcing >> DIRAC's bashrc: >> >> $ pip >> Traceback (most recent call last): >> File >> "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", >> line 7, in <module> >> from pip import main >> ImportError: cannot import name main >> >> Cheers >> Lukas >> >> Am 09/11/2018 um 15:16 schrieb Simon Fayer: >>> Hi Lukas, >>> >>> The easiest way of doing this is to treat the DIRAC python environment as >>> if it's a virtualenv and install whatever you need in to that (rather than >>> the other way around). >>> >>> # Install DIRAC UI & then start with a clean env... >>> # Get a subshell >>> bash >>> source bashrc >>> unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip >>> pip install -e t2kdm # or whatever >>> exit >>> source bashrc >>> # You should now have a python environment with your modules & DIRAC >>> >>> Regards, >>> Simon >>> >>> >>> On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote: >>>> Hello, >>>> >>>> I am trying to implement a Dirac backend for the software we use at T2K >>>> to access the grid data. This software is written in Python and now I am >>>> facing the problem that Dirac comes with its own Python distribution >>>> that does *not* play nicely with either the system Python or virtual >>>> environments. >>>> >>>> So my question is this: Is there a(n easy) way to make Dirac compatible >>>> with other Python software? Maybe a way to install the Dirac client in a >>>> regular virtual Python environment? >>>> >>>> Cheers >>>> Lukas >>>> >>
Hi Simon, that seems to work. I now have a working environment in which I can develop the DIRAC backend for t2kdm. Thanks! Lukas Am 12/11/2018 um 10:10 schrieb Simon Fayer:
Hi Lukas,
This is a bug in the bundling of the DIRAC externals: We've reported it back to the developers but it evidently hasn't been fixed yet. (The problem is that the gfal2 shell package is being installed in the python2.6 directory even if it's built for 2.7.
A workaround is to add the misplaced directory to the python path:
export PYTHONPATH=${PYTHONPATH}:${DIRAC}/Linux_x86_64_glibc-2.12/lib/python2.6/site-packages
Regards, Simon
On Fri, Nov 09, 2018 at 05:37:24PM +0000, Lukas Koch wrote:
Possibly related question:
After sourcing DIRAC's bashrc, the gfal-* commands no longer work:
$ gfal-ls Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/gfal-ls", line 24, in <module> from gfal2_util.shell import Gfal2Shell ImportError: No module named gfal2_util.shell
This is with both the original and my modified bashrc. Is that expected behaviour? Seems a bit strange since this is apparently the gfal command distributed with DIRAC.
Cheers Lukas
Am 09/11/2018 um 17:27 schrieb Lukas Koch:
Hi Simon,
your solution is much cleaner than mine, thanks! I appended these lines to the stock bashrc and it seems to work for now:
# Turn DIRAC into something resembling a virtualenv unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip export PYTHONNOUSERSITE=1
Cheers Lukas
Am 09/11/2018 um 17:20 schrieb Simon Fayer:
You could try disabling using the ~/.local folder by doing:
export PYTHONNOUSERSITE=1
That should make its behaviour more like a real virtualenv...
Regards, Simon
On Fri, Nov 09, 2018 at 05:15:13PM +0000, Lukas Koch wrote:
Ok, so it seems like it is trying to import the pip module from my ~/.local folder:
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3 import pip # directory /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip # /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo matches /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.py import pip # precompiled from /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo
Which is weird, since that path is not in my PYTHONPATH and both python and pip seem to come from the right place:
$ which python /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/python $ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
I guess it must be one of the default paths used by Python. It looks like it comes just before the DIRAC folders priority though:
$ python -c 'import sys; print sys.path' ['', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python27.zip', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/plat-linux2', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-tk', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-old', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-dynload', '/home/ppd/kdf77245/.local/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages/GSI-0.6.5-py2.7-linux-x86_64.egg']
Am 09/11/2018 um 16:55 schrieb Simon Fayer:
Hi Lukas,
I don't think SL6 should be substantially different to CentOS6. You don't happen to have a directory called "pip" in your CWD do you?
You could try running the following to see where it's finding the pip module from (i.e. to check it's using the one from the DIRAC bundle):
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3
Regards, Simon
On Fri, Nov 09, 2018 at 04:48:11PM +0000, Lukas Koch wrote: > Hello Simon, > > I did install it today with exactly the same command line as you quoted. > I do have locally installed python packages, but I assume those will not > affect the DIRAC Python. At least they should not, if I am to treat it > like a virtual environment. Anyway it seems like it is trying to use the > right pip: > > $ which pip > /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip > > And the PYTHONPATH looks reasonable as well: > > $ env | grep PYTHON > PYTHONUNBUFFERED=yes > PYTHONOPTIMIZE=x > PYTHONPATH=/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC: > > This is on a machine running SL6, by the way. Maybe that has something > to do with it. > > Cheers > Lukas > > Am 09/11/2018 um 16:12 schrieb Simon Fayer: >> Hi Lukas, >> >> Did you install DIRAC recently? Do you have any other python stuff >> installed in your user directory that may be interfering (--user installed >> pip or PYTHONPATH in ~/.bashrc)? >> >> I tried it from a clean install (on CentOS6) and it seems to work for me... >> >> $ mkdir testdir; cd testdir >> $ wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/d... >> $ chmod u+x dirac-install >> $ ./dirac-install -r v6r20p15 -i 27 -g v14r1 >> $ source bashrc >> $ unset REQUESTS_CA_BUNDLE SSL_CERT_DIR >> $ pip help >> >> Usage: >> pip <command> [options] >> >> ... >> >> Regards, >> Simon >> >> >> On Fri, Nov 09, 2018 at 03:46:19PM +0000, Lukas Koch wrote: >>> Hello Simon, >>> >>> I tried to do as you said, but pip does not seem to work after sourcing >>> DIRAC's bashrc: >>> >>> $ pip >>> Traceback (most recent call last): >>> File >>> "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", >>> line 7, in <module> >>> from pip import main >>> ImportError: cannot import name main >>> >>> Cheers >>> Lukas >>> >>> Am 09/11/2018 um 15:16 schrieb Simon Fayer: >>>> Hi Lukas, >>>> >>>> The easiest way of doing this is to treat the DIRAC python environment as >>>> if it's a virtualenv and install whatever you need in to that (rather than >>>> the other way around). >>>> >>>> # Install DIRAC UI & then start with a clean env... >>>> # Get a subshell >>>> bash >>>> source bashrc >>>> unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip >>>> pip install -e t2kdm # or whatever >>>> exit >>>> source bashrc >>>> # You should now have a python environment with your modules & DIRAC >>>> >>>> Regards, >>>> Simon >>>> >>>> >>>> On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote: >>>>> Hello, >>>>> >>>>> I am trying to implement a Dirac backend for the software we use at T2K >>>>> to access the grid data. This software is written in Python and now I am >>>>> facing the problem that Dirac comes with its own Python distribution >>>>> that does *not* play nicely with either the system Python or virtual >>>>> environments. >>>>> >>>>> So my question is this: Is there a(n easy) way to make Dirac compatible >>>>> with other Python software? Maybe a way to install the Dirac client in a >>>>> regular virtual Python environment? >>>>> >>>>> Cheers >>>>> Lukas >>>>> >>> >
-- Lukas KOCH Particle Physics Department STFC Rutherford Appleton Laboratory Mail: lukas.koch@stfc.ac.uk Tel: +44 (0) 1235 44 x6265
I guess the problem is that the DIRAC Python is handled like a system Python, so the --user installs take precedence over the "system" packages. I managed to "fix" this by changing one line in the bashrc to: ( echo $PYTHONPATH | grep -q $DIRAC ) || export PYTHONPATH=$DIRAC:$DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages/:$PYTHONPATH Now the "system" packages have higher priority than the --user packages. But think the "proper" way to deal with this would probably be to actually put the DIRAC install into a proper virtualenv. I assume that might be a bit of work to implement though. Cheers Lukas Am 09/11/2018 um 17:15 schrieb Lukas Koch:
Ok, so it seems like it is trying to import the pip module from my ~/.local folder:
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3 import pip # directory /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip # /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo matches /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.py import pip # precompiled from /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo
Which is weird, since that path is not in my PYTHONPATH and both python and pip seem to come from the right place:
$ which python /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/python $ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
I guess it must be one of the default paths used by Python. It looks like it comes just before the DIRAC folders priority though:
$ python -c 'import sys; print sys.path' ['', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python27.zip', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/plat-linux2', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-tk', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-old', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-dynload', '/home/ppd/kdf77245/.local/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages/GSI-0.6.5-py2.7-linux-x86_64.egg']
Am 09/11/2018 um 16:55 schrieb Simon Fayer:
Hi Lukas,
I don't think SL6 should be substantially different to CentOS6. You don't happen to have a directory called "pip" in your CWD do you?
You could try running the following to see where it's finding the pip module from (i.e. to check it's using the one from the DIRAC bundle):
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3
Regards, Simon
On Fri, Nov 09, 2018 at 04:48:11PM +0000, Lukas Koch wrote:
Hello Simon,
I did install it today with exactly the same command line as you quoted. I do have locally installed python packages, but I assume those will not affect the DIRAC Python. At least they should not, if I am to treat it like a virtual environment. Anyway it seems like it is trying to use the right pip:
$ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
And the PYTHONPATH looks reasonable as well:
$ env | grep PYTHON PYTHONUNBUFFERED=yes PYTHONOPTIMIZE=x PYTHONPATH=/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC:
This is on a machine running SL6, by the way. Maybe that has something to do with it.
Cheers Lukas
Am 09/11/2018 um 16:12 schrieb Simon Fayer:
Hi Lukas,
Did you install DIRAC recently? Do you have any other python stuff installed in your user directory that may be interfering (--user installed pip or PYTHONPATH in ~/.bashrc)?
I tried it from a clean install (on CentOS6) and it seems to work for me...
$ mkdir testdir; cd testdir $ wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/d... $ chmod u+x dirac-install $ ./dirac-install -r v6r20p15 -i 27 -g v14r1 $ source bashrc $ unset REQUESTS_CA_BUNDLE SSL_CERT_DIR $ pip help
Usage: pip <command> [options]
...
Regards, Simon
On Fri, Nov 09, 2018 at 03:46:19PM +0000, Lukas Koch wrote:
Hello Simon,
I tried to do as you said, but pip does not seem to work after sourcing DIRAC's bashrc:
$ pip Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", line 7, in <module> from pip import main ImportError: cannot import name main
Cheers Lukas
Am 09/11/2018 um 15:16 schrieb Simon Fayer:
Hi Lukas,
The easiest way of doing this is to treat the DIRAC python environment as if it's a virtualenv and install whatever you need in to that (rather than the other way around).
# Install DIRAC UI & then start with a clean env... # Get a subshell bash source bashrc unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip pip install -e t2kdm # or whatever exit source bashrc # You should now have a python environment with your modules & DIRAC
Regards, Simon
On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote: > Hello, > > I am trying to implement a Dirac backend for the software we use at T2K > to access the grid data. This software is written in Python and now I am > facing the problem that Dirac comes with its own Python distribution > that does *not* play nicely with either the system Python or virtual > environments. > > So my question is this: Is there a(n easy) way to make Dirac compatible > with other Python software? Maybe a way to install the Dirac client in a > regular virtual Python environment? > > Cheers > Lukas >
-- Lukas KOCH Particle Physics Department STFC Rutherford Appleton Laboratory Mail: lukas.koch@stfc.ac.uk Tel: +44 (0) 1235 44 x6265
Hi Lukas, On Fri, Nov 09, 2018 at 05:23:04PM +0000, Lukas Koch wrote:
But think the "proper" way to deal with this would probably be to actually put the DIRAC install into a proper virtualenv. I assume that might be a bit of work to implement though.
Yes, the problem is that DIRAC has quite a few dependencies on various grid utils (which would have to be distributed outside of pip) and their python bindings (things like gfal2); all of these have to be built together to ensure compatibility. The large bundle approach does seem to work very well on the grid, but yes, accessing/using the API can be a bit troublesome at times (We have similar problems with a couple of projects we wrote for other VOs). Regards, Simon
participants (2)
-
Lukas Koch
-
Simon Fayer