Still this is the programming equivalent of what the NHS calls a "never event" ;-) On 21 March 2016 at 09:41, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
And asking around, it is fixed already.
https://github.com/DIRACGrid/DIRAC/commit/536802ae2fbd4d891be3c5bf947a87aa3d...
Cheers, Raja.
On 21/03/16 05:27, Raja Nandakumar wrote:
Hi Steve,
Thanks - it is funny, I do not find it in the github repository of the file at
https://github.com/DIRACGrid/DIRAC/blob/v6r14p19/DataManagementSystem/Client...
I will ask.
Cheers, Raja.
On 19/03/16 16:33, Steve Lloyd wrote:
Hi Raja, It's DIRAC/DataManagementSystem/Client/DataManager.py line 1640. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 15:21, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Which line is it? I did not find such an obviously strange piece of code. If you can point it out to me, I will be happy to feed it back to the DIRAC developers.
Cheers, Raja.
On 18/03/16 14:51, Steve Lloyd wrote:
It was some time ago - I'm going to upload some more so I'll see what happens with them.
By the way I thought I understood Python till I saw this (in DataManager.py):
# # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo )
Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> On 17 Mar 2016, at 17:26, Daniela Bauer > <daniela.bauer.grid@googlemail.com> wrote: > > Hi Steve, > > I can't quite reproduce your problem. > I do the following: > I upload a file to the Imperial SE using dirac-dms-add-file > lx05:2016_Jan_18_1437 :~] dirac-dms-add-file > /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data > UKI-LT2-IC-HEP-disk > > Uploading /gridpp/user/d/daniela.bauer/test100mb.data > Successfully uploaded file to UKI-LT2-IC-HEP-disk > > then I replicate it to QMUL: > lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn > /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk > {'Failed': {}, > 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': > {'register': 0.09097790718078613, > > 'replicate': 17.55987310409546}}} > > > then I ask for the accessURL: > lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL > /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk > {'Failed': {}, > 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': > > 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}} > > and > > lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL > /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk > {'Failed': {}, > 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': > > 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}} > > > which are both correct and I can use it with lcg-ls (not in same > terminal though): > lx04:~ :~] lcg-ls > > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data > /gridpp/gridpp/user/d/daniela.bauer/test100mb.data > > lx04:~ :~] lcg-ls > > srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data > > /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data > > When you uploaded your files, did you mix and match lcg and dirac > tools ? > > Regards, > Daniela > > On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> > wrote: >> >> Hi Steve, >> >> Interesting. Unfortunately, I think that the attitude taken >> currently in the >> DIRAC DMS system is that the users do not care where they get their >> data >> from. So, the DIRAC tool does not currently provide any option to >> choose a >> specific SE. >> >> In previous such cases, I have taken the path of hacking into DIRAC >> to force >> it to do exactly what I want. In this case, you may want to look at >> the >> function "getFile" which is defined in for example >> >> >> /cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py >> >> It looks easy enough to implement on a private script ... :) >> >> Cheers, >> Raja. >> >> >> On 17/03/16 10:14, Steve Lloyd wrote: >>> >>> >>> Hi, >>> Unfortunately it appears that Raja's method does not work at >>> QMUL. At >>> QMUL you have to use the surl returned by dirac-dms-lfn-replicas >>> and at >>> Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there >>> 2 surls >>> in the first place?) Is there a Dirac tool to retrieve a particular >>> replica? >>> Cheers >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Prof Steve Lloyd >>> Head of School of Physics and Astronomy >>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>>> On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> >>>> wrote: >>>> >>>> Hi Steve, >>>> >>>> In general, I agree with Raja, though: not letting the DIRAC tools >>>> handle >>>> files that are managed via DIRAC causes problems (like the one you >>>> just >>>> encountered), as the DIRAC namespace is not the SE namespace in >>>> general >>>> (just as with LFCs). >>>> (In your case, Raja's method to retrieve SURLs is the best one.) >>>> >>>> Sam >>>> >>>> On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> >>>> wrote: >>>> Hi Raja, >>>> Thanks - at QMUL the two surls are the same (apart from a //) so >>>> it >>>> worked whereas at Glasgow they are different so it didn't. It was >>>> confusing >>>> that it worked at QMUL but not at Glasgow. I want to do this in a >>>> running >>>> job to get a particular replica to measure the bandwidth which is >>>> why I'm >>>> not using the Dirac tools. >>>> Cheers >>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Prof Steve Lloyd >>>> Head of School of Physics and Astronomy >>>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >>>>> On 16 Mar 2016, at 15:32, Raja Nandakumar >>>>> <raja.nandakumar@cern.ch> >>>>> wrote: >>>>> >>>>> Hi Steve, >>>>> >>>>> The best way is to use DIRAC tools directly for downloading the >>>>> file. >>>>> The tool here is : >>>>> >>>>>>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data >>>>> >>>>> >>>>> >>>>> This command should do all the needed steps under the hood, >>>>> picking any >>>>> available SE at random to get the file. >>>>> >>>>> But if you want to go the lcg-cp way, note that the surl you pass >>>>> to >>>>> lcg-cp is incorrect and the cause of your error. To get the >>>>> correct surl, >>>>> you need to do the following : >>>>> >>>>> 1. Get the list of replicas (as you did) >>>>>>> >>>>>>> >>>>>>> dirac-dms-lfn-replicas >>>>>>> /gridpp/user/s/steve.lloyd/test100mb.data >>>>> >>>>> >>>>> LFN StorageElement >>>>> URL >>>>> >>>>> >>>>> ======================================================================================================================================================================= >>>>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk >>>>> >>>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> >>>>> UKI-SCOTGRID-GLASGOW-disk >>>>> >>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> UKI-LT2-QMUL2-disk >>>>> >>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> >>>>> >>>>> 2. Get the accessURL for any one of the replicas : >>>>>>> >>>>>>> >>>>>>> dirac-dms-lfn-accessURL >>>>>>> /gridpp/user/s/steve.lloyd/test100mb.data >>>>>>> UKI-LT2-IC-HEP-disk >>>>> >>>>> >>>>> {'Failed': {}, >>>>> 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': >>>>> >>>>> 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} >>>>> >>>>> 3. Pass it to lcg-cp : >>>>>>> >>>>>>> >>>>>>> lcg-cp -v --vo gridpp >>>>>>> >>>>>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>>>>>> test.data >>>>> >>>>> >>>>> >>>>> >>>>> Hoping this helps, >>>>> >>>>> Cheers, >>>>> Raja. >>>>> >>>>> On 16/03/16 15:12, Steve Lloyd wrote: >>>>>> >>>>>> >>>>>> Hi, >>>>>> Anyone know what's wrong here. (Presumably the file got >>>>>> deleted but >>>>>> the catalogue wasn't updated). >>>>>> Cheers >>>>>> >>>>>> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >>>>>> /gridpp/user/s/steve.lloyd/test100mb.data >>>>>> LFN StorageElement >>>>>> URL >>>>>> >>>>>> >>>>>> ====================================================================================================================================================================== >>>>>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >>>>>> >>>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> >>>>>> UKI-SCOTGRID-GLASGOW-disk >>>>>> >>>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> >>>>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>>>> >>>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> file:t100.out >>>>>> Using grid catalog type: UNKNOWN >>>>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>>>> VO name: gridpp >>>>>> Checksum type: None >>>>>> Trying SURL >>>>>> >>>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> ... >>>>>> Source SE type: SRMv2 >>>>>> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >>>>>> Source URL: >>>>>> >>>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> File size: 100000000 >>>>>> Source URL for copy: >>>>>> >>>>>> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >>>>>> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >>>>>> # streams: 1 >>>>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>>>> >>>>>> Transfer took 9010 ms >>>>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>>>> >>>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> file:t100.out >>>>>> Using grid catalog type: UNKNOWN >>>>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>>>> VO name: gridpp >>>>>> Checksum type: None >>>>>> Trying SURL >>>>>> >>>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> ... >>>>>> [SE][Ls][SRM_INVALID_PATH] No such file or directory >>>>>> lcg_cp: No such file or directory >>>>>> >>>>>> >>>>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> Prof Steve Lloyd >>>>>> Head of School of Physics and Astronomy >>>>>> Queen Mary University of London, Mile End Road, London E1 4NS, >>>>>> UK >>>>>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>>>>> >>>>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> >>>>>> >>>>> >>>>> -- >>>>> _______________________________________________ >>>>> Gridpp-Dirac-Users mailing list >>>>> Gridpp-Dirac-Users@imperial.ac.uk >>>>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >>>> >>>> >>>> >>>> >>>> -- >>>> _______________________________________________ >>>> Gridpp-Dirac-Users mailing list >>>> Gridpp-Dirac-Users@imperial.ac.uk >>>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >>>> -- >>>> _______________________________________________ >>>> Gridpp-Dirac-Users mailing list >>>> Gridpp-Dirac-Users@imperial.ac.uk >>>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >>> >>> >>> >>> >> >> -- >> _______________________________________________ >> Gridpp-Dirac-Users mailing list >> Gridpp-Dirac-Users@imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users > > > > > -- > Sent from the pit of despair > > ----------------------------------------------------------- > daniela.bauer@imperial.ac.uk > HEP Group/Physics Dep > Imperial College > London, SW7 2BW > Tel: +44-(0)20-75947810 > http://www.hep.ph.ic.ac.uk/~dbauer/ > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/