Possibly related question: After sourcing DIRAC's bashrc, the gfal-* commands no longer work: $ gfal-ls Traceback (most recent call last): File "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/gfal-ls", line 24, in <module> from gfal2_util.shell import Gfal2Shell ImportError: No module named gfal2_util.shell This is with both the original and my modified bashrc. Is that expected behaviour? Seems a bit strange since this is apparently the gfal command distributed with DIRAC. Cheers Lukas Am 09/11/2018 um 17:27 schrieb Lukas Koch:
Hi Simon,
your solution is much cleaner than mine, thanks! I appended these lines to the stock bashrc and it seems to work for now:
# Turn DIRAC into something resembling a virtualenv unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip export PYTHONNOUSERSITE=1
Cheers Lukas
Am 09/11/2018 um 17:20 schrieb Simon Fayer:
You could try disabling using the ~/.local folder by doing:
export PYTHONNOUSERSITE=1
That should make its behaviour more like a real virtualenv...
Regards, Simon
On Fri, Nov 09, 2018 at 05:15:13PM +0000, Lukas Koch wrote:
Ok, so it seems like it is trying to import the pip module from my ~/.local folder:
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3 import pip # directory /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip # /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo matches /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.py import pip # precompiled from /home/ppd/kdf77245/.local/lib/python2.7/site-packages/pip/__init__.pyo
Which is weird, since that path is not in my PYTHONPATH and both python and pip seem to come from the right place:
$ which python /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/python $ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
I guess it must be one of the default paths used by Python. It looks like it comes just before the DIRAC folders priority though:
$ python -c 'import sys; print sys.path' ['', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python27.zip', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/plat-linux2', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-tk', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-old', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/lib-dynload', '/home/ppd/kdf77245/.local/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages', '/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/lib/python2.7/site-packages/GSI-0.6.5-py2.7-linux-x86_64.egg']
Am 09/11/2018 um 16:55 schrieb Simon Fayer:
Hi Lukas,
I don't think SL6 should be substantially different to CentOS6. You don't happen to have a directory called "pip" in your CWD do you?
You could try running the following to see where it's finding the pip module from (i.e. to check it's using the one from the DIRAC bundle):
$ python -v -c "import pip" 2>&1 | grep pip | head -n 3
Regards, Simon
On Fri, Nov 09, 2018 at 04:48:11PM +0000, Lukas Koch wrote:
Hello Simon,
I did install it today with exactly the same command line as you quoted. I do have locally installed python packages, but I assume those will not affect the DIRAC Python. At least they should not, if I am to treat it like a virtual environment. Anyway it seems like it is trying to use the right pip:
$ which pip /opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip
And the PYTHONPATH looks reasonable as well:
$ env | grep PYTHON PYTHONUNBUFFERED=yes PYTHONOPTIMIZE=x PYTHONPATH=/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC:
This is on a machine running SL6, by the way. Maybe that has something to do with it.
Cheers Lukas
Am 09/11/2018 um 16:12 schrieb Simon Fayer:
Hi Lukas,
Did you install DIRAC recently? Do you have any other python stuff installed in your user directory that may be interfering (--user installed pip or PYTHONPATH in ~/.bashrc)?
I tried it from a clean install (on CentOS6) and it seems to work for me...
$ mkdir testdir; cd testdir $ wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/d... $ chmod u+x dirac-install $ ./dirac-install -r v6r20p15 -i 27 -g v14r1 $ source bashrc $ unset REQUESTS_CA_BUNDLE SSL_CERT_DIR $ pip help
Usage: pip <command> [options]
...
Regards, Simon
On Fri, Nov 09, 2018 at 03:46:19PM +0000, Lukas Koch wrote: > Hello Simon, > > I tried to do as you said, but pip does not seem to work after sourcing > DIRAC's bashrc: > > $ pip > Traceback (most recent call last): > File > "/opt/ppd/scratch/kdf77245/t2k_software/t2kGrid/t2kdm/DIRAC/Linux_x86_64_glibc-2.12/bin/pip", > line 7, in <module> > from pip import main > ImportError: cannot import name main > > Cheers > Lukas > > Am 09/11/2018 um 15:16 schrieb Simon Fayer: >> Hi Lukas, >> >> The easiest way of doing this is to treat the DIRAC python environment as >> if it's a virtualenv and install whatever you need in to that (rather than >> the other way around). >> >> # Install DIRAC UI & then start with a clean env... >> # Get a subshell >> bash >> source bashrc >> unset REQUESTS_CA_BUNDLE SSL_CERT_DIR # These upset pip >> pip install -e t2kdm # or whatever >> exit >> source bashrc >> # You should now have a python environment with your modules & DIRAC >> >> Regards, >> Simon >> >> >> On Fri, Nov 09, 2018 at 01:16:30PM +0000, Lukas Koch wrote: >>> Hello, >>> >>> I am trying to implement a Dirac backend for the software we use at T2K >>> to access the grid data. This software is written in Python and now I am >>> facing the problem that Dirac comes with its own Python distribution >>> that does *not* play nicely with either the system Python or virtual >>> environments. >>> >>> So my question is this: Is there a(n easy) way to make Dirac compatible >>> with other Python software? Maybe a way to install the Dirac client in a >>> regular virtual Python environment? >>> >>> Cheers >>> Lukas >>> >
-- Lukas KOCH Particle Physics Department STFC Rutherford Appleton Laboratory Mail: lukas.koch@stfc.ac.uk Tel: +44 (0) 1235 44 x6265