And asking around, it is fixed already. https://github.com/DIRACGrid/DIRAC/commit/536802ae2fbd4d891be3c5bf947a87aa3d... Cheers, Raja. On 21/03/16 05:27, Raja Nandakumar wrote:
Hi Steve,
Thanks - it is funny, I do not find it in the github repository of the file at
https://github.com/DIRACGrid/DIRAC/blob/v6r14p19/DataManagementSystem/Client...
I will ask.
Cheers, Raja.
On 19/03/16 16:33, Steve Lloyd wrote:
Hi Raja, It's DIRAC/DataManagementSystem/Client/DataManager.py line 1640. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 15:21, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Which line is it? I did not find such an obviously strange piece of code. If you can point it out to me, I will be happy to feed it back to the DIRAC developers.
Cheers, Raja.
On 18/03/16 14:51, Steve Lloyd wrote:
It was some time ago - I'm going to upload some more so I'll see what happens with them.
By the way I thought I understood Python till I saw this (in DataManager.py):
# # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo )
Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote: > Hi Steve, > > Interesting. Unfortunately, I think that the attitude taken currently in the > DIRAC DMS system is that the users do not care where they get their data > from. So, the DIRAC tool does not currently provide any option to choose a > specific SE. > > In previous such cases, I have taken the path of hacking into DIRAC to force > it to do exactly what I want. In this case, you may want to look at the > function "getFile" which is defined in for example > > /cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py > > It looks easy enough to implement on a private script ... :) > > Cheers, > Raja. > > > On 17/03/16 10:14, Steve Lloyd wrote: >> >> Hi, >> Unfortunately it appears that Raja's method does not work at QMUL. At >> QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at >> Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls >> in the first place?) Is there a Dirac tool to retrieve a particular replica? >> Cheers >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Prof Steve Lloyd >> Head of School of Physics and Astronomy >> Queen Mary University of London, Mile End Road, London E1 4NS, UK >> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote: >>> >>> Hi Steve, >>> >>> In general, I agree with Raja, though: not letting the DIRAC tools handle >>> files that are managed via DIRAC causes problems (like the one you just >>> encountered), as the DIRAC namespace is not the SE namespace in general >>> (just as with LFCs). >>> (In your case, Raja's method to retrieve SURLs is the best one.) >>> >>> Sam >>> >>> On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: >>> Hi Raja, >>> Thanks - at QMUL the two surls are the same (apart from a //) so it >>> worked whereas at Glasgow they are different so it didn't. It was confusing >>> that it worked at QMUL but not at Glasgow. I want to do this in a running >>> job to get a particular replica to measure the bandwidth which is why I'm >>> not using the Dirac tools. >>> Cheers >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Prof Steve Lloyd >>> Head of School of Physics and Astronomy >>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>>> On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> >>>> wrote: >>>> >>>> Hi Steve, >>>> >>>> The best way is to use DIRAC tools directly for downloading the file. >>>> The tool here is : >>>> >>>>>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data >>>> >>>> >>>> This command should do all the needed steps under the hood, picking any >>>> available SE at random to get the file. >>>> >>>> But if you want to go the lcg-cp way, note that the surl you pass to >>>> lcg-cp is incorrect and the cause of your error. To get the correct surl, >>>> you need to do the following : >>>> >>>> 1. Get the list of replicas (as you did) >>>>>> >>>>>> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data >>>> >>>> LFN StorageElement URL >>>> >>>> ======================================================================================================================================================================= >>>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk >>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> UKI-SCOTGRID-GLASGOW-disk >>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> UKI-LT2-QMUL2-disk >>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> >>>> >>>> 2. Get the accessURL for any one of the replicas : >>>>>> >>>>>> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >>>>>> UKI-LT2-IC-HEP-disk >>>> >>>> {'Failed': {}, >>>> 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': >>>> 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} >>>> >>>> 3. Pass it to lcg-cp : >>>>>> >>>>>> lcg-cp -v --vo gridpp >>>>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>>>>> test.data >>>> >>>> >>>> >>>> Hoping this helps, >>>> >>>> Cheers, >>>> Raja. >>>> >>>> On 16/03/16 15:12, Steve Lloyd wrote: >>>>> >>>>> Hi, >>>>> Anyone know what's wrong here. (Presumably the file got deleted but >>>>> the catalogue wasn't updated). >>>>> Cheers >>>>> >>>>> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >>>>> /gridpp/user/s/steve.lloyd/test100mb.data >>>>> LFN StorageElement URL >>>>> >>>>> ====================================================================================================================================================================== >>>>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> UKI-SCOTGRID-GLASGOW-disk >>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> >>>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> file:t100.out >>>>> Using grid catalog type: UNKNOWN >>>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>>> VO name: gridpp >>>>> Checksum type: None >>>>> Trying SURL >>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> ... >>>>> Source SE type: SRMv2 >>>>> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >>>>> Source URL: >>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> File size: 100000000 >>>>> Source URL for copy: >>>>> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >>>>> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >>>>> # streams: 1 >>>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>>> >>>>> Transfer took 9010 ms >>>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> file:t100.out >>>>> Using grid catalog type: UNKNOWN >>>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>>> VO name: gridpp >>>>> Checksum type: None >>>>> Trying SURL >>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> ... >>>>> [SE][Ls][SRM_INVALID_PATH] No such file or directory >>>>> lcg_cp: No such file or directory >>>>> >>>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>> Prof Steve Lloyd >>>>> Head of School of Physics and Astronomy >>>>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>>>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>> >>>>> >>>> >>>> -- >>>> _______________________________________________ >>>> Gridpp-Dirac-Users mailing list >>>> Gridpp-Dirac-Users@imperial.ac.uk >>>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >>> >>> >>> >>> -- >>> _______________________________________________ >>> Gridpp-Dirac-Users mailing list >>> Gridpp-Dirac-Users@imperial.ac.uk >>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >>> -- >>> _______________________________________________ >>> Gridpp-Dirac-Users mailing list >>> Gridpp-Dirac-Users@imperial.ac.uk >>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >> >> >> > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users