Now, when I stand up at the gridpp meeting and complain about the difficulties of installing/administering/debugging/fixing dirac, at least you know what I am talking about :-D Regards, Daniela On 18 March 2016 at 14:51, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
It was some time ago - I'm going to upload some more so I'll see what happens with them.
By the way I thought I understood Python till I saw this (in DataManager.py):
# # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo )
Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote:
Hi Steve,
In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.)
Sam
On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> > wrote: > > Hi Steve, > > The best way is to use DIRAC tools directly for downloading the file. > The tool here is : > >>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data > > > This command should do all the needed steps under the hood, picking any > available SE at random to get the file. > > But if you want to go the lcg-cp way, note that the surl you pass to > lcg-cp is incorrect and the cause of your error. To get the correct surl, > you need to do the following : > > 1. Get the list of replicas (as you did) >>> >>> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data > > LFN StorageElement URL > > ======================================================================================================================================================================= > /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk > srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > UKI-SCOTGRID-GLASGOW-disk > srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > UKI-LT2-QMUL2-disk > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > > > 2. Get the accessURL for any one of the replicas : >>> >>> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >>> UKI-LT2-IC-HEP-disk > > {'Failed': {}, > 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': > 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} > > 3. Pass it to lcg-cp : >>> >>> lcg-cp -v --vo gridpp >>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>> test.data > > > > Hoping this helps, > > Cheers, > Raja. > > On 16/03/16 15:12, Steve Lloyd wrote: >> >> Hi, >> Anyone know what's wrong here. (Presumably the file got deleted but >> the catalogue wasn't updated). >> Cheers >> >> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >> /gridpp/user/s/steve.lloyd/test100mb.data >> LFN StorageElement URL >> >> ====================================================================================================================================================================== >> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> UKI-SCOTGRID-GLASGOW-disk >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> >> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> file:t100.out >> Using grid catalog type: UNKNOWN >> Using grid catalog : prod-lfc-atlas.cern.ch >> VO name: gridpp >> Checksum type: None >> Trying SURL >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> ... >> Source SE type: SRMv2 >> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >> Source URL: >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> File size: 100000000 >> Source URL for copy: >> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >> # streams: 1 >> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >> >> Transfer took 9010 ms >> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> file:t100.out >> Using grid catalog type: UNKNOWN >> Using grid catalog : prod-lfc-atlas.cern.ch >> VO name: gridpp >> Checksum type: None >> Trying SURL >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> ... >> [SE][Ls][SRM_INVALID_PATH] No such file or directory >> lcg_cp: No such file or directory >> >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Prof Steve Lloyd >> Head of School of Physics and Astronomy >> Queen Mary University of London, Mile End Road, London E1 4NS, UK >> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/