Hi Steve, Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE. In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example /cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py It looks easy enough to implement on a private script ... :) Cheers, Raja. On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote:
Hi Steve,
In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.)
Sam
On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
The best way is to use DIRAC tools directly for downloading the file. The tool here is :
dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data
This command should do all the needed steps under the hood, picking any available SE at random to get the file.
But if you want to go the lcg-cp way, note that the surl you pass to lcg-cp is incorrect and the cause of your error. To get the correct surl, you need to do the following :
1. Get the list of replicas (as you did)
dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL ======================================================================================================================================================================= /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
2. Get the accessURL for any one of the replicas :
dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}}
3. Pass it to lcg-cp :
lcg-cp -v --vo gridpp srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data test.data
Hoping this helps,
Cheers, Raja.
On 16/03/16 15:12, Steve Lloyd wrote:
Hi, Anyone know what's wrong here. (Presumably the file got deleted but the catalogue wasn't updated). Cheers
[lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL ====================================================================================================================================================================== /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
[lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... Source SE type: SRMv2 Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 Source URL: srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data File size: 100000000 Source URL for copy: gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out # streams: 1 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst
Transfer took 9010 ms [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... [SE][Ls][SRM_INVALID_PATH] No such file or directory lcg_cp: No such file or directory
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users