Hi, Anyone know what's wrong here. (Presumably the file got deleted but the catalogue wasn't updated). Cheers [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL ====================================================================================================================================================================== /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... Source SE type: SRMv2 Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 Source URL: srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data File size: 100000000 Source URL for copy: gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out # streams: 1 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst Transfer took 9010 ms [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... [SE][Ls][SRM_INVALID_PATH] No such file or directory lcg_cp: No such file or directory +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Hi Steve, The best way is to use DIRAC tools directly for downloading the file. The tool here is :
dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data
This command should do all the needed steps under the hood, picking any available SE at random to get the file. But if you want to go the lcg-cp way, note that the surl you pass to lcg-cp is incorrect and the cause of your error. To get the correct surl, you need to do the following : 1. Get the list of replicas (as you did)
dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL ======================================================================================================================================================================= /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
2. Get the accessURL for any one of the replicas :
dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}}
3. Pass it to lcg-cp :
lcg-cp -v --vo gridpp srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data test.data
Hoping this helps, Cheers, Raja. On 16/03/16 15:12, Steve Lloyd wrote:
Hi, Anyone know what's wrong here. (Presumably the file got deleted but the catalogue wasn't updated). Cheers
[lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL ====================================================================================================================================================================== /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
[lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... Source SE type: SRMv2 Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 Source URL: srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data File size: 100000000 Source URL for copy: gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out # streams: 1 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst
Transfer took 9010 ms [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... [SE][Ls][SRM_INVALID_PATH] No such file or directory lcg_cp: No such file or directory
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
The best way is to use DIRAC tools directly for downloading the file. The tool here is :
dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data
This command should do all the needed steps under the hood, picking any available SE at random to get the file.
But if you want to go the lcg-cp way, note that the surl you pass to lcg-cp is incorrect and the cause of your error. To get the correct surl, you need to do the following :
1. Get the list of replicas (as you did)
dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL ======================================================================================================================================================================= /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
2. Get the accessURL for any one of the replicas :
dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}}
3. Pass it to lcg-cp :
lcg-cp -v --vo gridpp srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data test.data
Hoping this helps,
Cheers, Raja.
On 16/03/16 15:12, Steve Lloyd wrote:
Hi, Anyone know what's wrong here. (Presumably the file got deleted but the catalogue wasn't updated). Cheers
[lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL ====================================================================================================================================================================== /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
[lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... Source SE type: SRMv2 Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 Source URL: srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data File size: 100000000 Source URL for copy: gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out # streams: 1 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst
Transfer took 9010 ms [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... [SE][Ls][SRM_INVALID_PATH] No such file or directory lcg_cp: No such file or directory
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Hi Steve, In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.) Sam On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
The best way is to use DIRAC tools directly for downloading the file. The tool here is :
dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data
This command should do all the needed steps under the hood, picking any available SE at random to get the file.
But if you want to go the lcg-cp way, note that the surl you pass to lcg-cp is incorrect and the cause of your error. To get the correct surl, you need to do the following :
1. Get the list of replicas (as you did)
dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL
=======================================================================================================================================================================
/gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk srm:// gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm:// svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm:// se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
2. Get the accessURL for any one of the replicas :
dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': 'srm:// gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data' }}
3. Pass it to lcg-cp :
lcg-cp -v --vo gridpp srm:// gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data test.data
Hoping this helps,
Cheers, Raja.
On 16/03/16 15:12, Steve Lloyd wrote:
Hi, Anyone know what's wrong here. (Presumably the file got deleted but the catalogue wasn't updated). Cheers
[lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL
======================================================================================================================================================================
/gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm:// se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm:// svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
[lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm:// se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm:// se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... Source SE type: SRMv2 Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 Source URL: srm:// se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data File size: 100000000 Source URL for copy: gsiftp:// se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out # streams: 1 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst
Transfer took 9010 ms [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm:// svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm:// svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... [SE][Ls][SRM_INVALID_PATH] No such file or directory lcg_cp: No such file or directory
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote:
Hi Steve,
In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.)
Sam
On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
The best way is to use DIRAC tools directly for downloading the file. The tool here is :
dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data
This command should do all the needed steps under the hood, picking any available SE at random to get the file.
But if you want to go the lcg-cp way, note that the surl you pass to lcg-cp is incorrect and the cause of your error. To get the correct surl, you need to do the following :
1. Get the list of replicas (as you did)
dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL ======================================================================================================================================================================= /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
2. Get the accessURL for any one of the replicas :
dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}}
3. Pass it to lcg-cp :
lcg-cp -v --vo gridpp srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data test.data
Hoping this helps,
Cheers, Raja.
On 16/03/16 15:12, Steve Lloyd wrote:
Hi, Anyone know what's wrong here. (Presumably the file got deleted but the catalogue wasn't updated). Cheers
[lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL ====================================================================================================================================================================== /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
[lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... Source SE type: SRMv2 Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 Source URL: srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data File size: 100000000 Source URL for copy: gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out # streams: 1 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst
Transfer took 9010 ms [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... [SE][Ls][SRM_INVALID_PATH] No such file or directory lcg_cp: No such file or directory
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Hi Steve, Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE. In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example /cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py It looks easy enough to implement on a private script ... :) Cheers, Raja. On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote:
Hi Steve,
In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.)
Sam
On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
The best way is to use DIRAC tools directly for downloading the file. The tool here is :
dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data
This command should do all the needed steps under the hood, picking any available SE at random to get the file.
But if you want to go the lcg-cp way, note that the surl you pass to lcg-cp is incorrect and the cause of your error. To get the correct surl, you need to do the following :
1. Get the list of replicas (as you did)
dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL ======================================================================================================================================================================= /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
2. Get the accessURL for any one of the replicas :
dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}}
3. Pass it to lcg-cp :
lcg-cp -v --vo gridpp srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data test.data
Hoping this helps,
Cheers, Raja.
On 16/03/16 15:12, Steve Lloyd wrote:
Hi, Anyone know what's wrong here. (Presumably the file got deleted but the catalogue wasn't updated). Cheers
[lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL ====================================================================================================================================================================== /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
[lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... Source SE type: SRMv2 Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 Source URL: srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data File size: 100000000 Source URL for copy: gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out # streams: 1 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst
Transfer took 9010 ms [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... [SE][Ls][SRM_INVALID_PATH] No such file or directory lcg_cp: No such file or directory
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Hi Steve, I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613, 'replicate': 17.55987310409546}}} then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}} and lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}} which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data When you uploaded your files, did you mix and match lcg and dirac tools ? Regards, Daniela On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote:
Hi Steve,
In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.)
Sam
On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
The best way is to use DIRAC tools directly for downloading the file. The tool here is :
dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data
This command should do all the needed steps under the hood, picking any available SE at random to get the file.
But if you want to go the lcg-cp way, note that the surl you pass to lcg-cp is incorrect and the cause of your error. To get the correct surl, you need to do the following :
1. Get the list of replicas (as you did)
dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data
LFN StorageElement URL
======================================================================================================================================================================= /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
2. Get the accessURL for any one of the replicas :
dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk
{'Failed': {}, 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}}
3. Pass it to lcg-cp :
lcg-cp -v --vo gridpp srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data test.data
Hoping this helps,
Cheers, Raja.
On 16/03/16 15:12, Steve Lloyd wrote:
Hi, Anyone know what's wrong here. (Presumably the file got deleted but the catalogue wasn't updated). Cheers
[lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL
====================================================================================================================================================================== /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
[lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... Source SE type: SRMv2 Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 Source URL: srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data File size: 100000000 Source URL for copy: gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out # streams: 1 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst
Transfer took 9010 ms [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... [SE][Ls][SRM_INVALID_PATH] No such file or directory lcg_cp: No such file or directory
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote:
Hi Steve,
In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.)
Sam
On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
The best way is to use DIRAC tools directly for downloading the file. The tool here is :
> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data
This command should do all the needed steps under the hood, picking any available SE at random to get the file.
But if you want to go the lcg-cp way, note that the surl you pass to lcg-cp is incorrect and the cause of your error. To get the correct surl, you need to do the following :
1. Get the list of replicas (as you did)
> > dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data
LFN StorageElement URL
======================================================================================================================================================================= /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
2. Get the accessURL for any one of the replicas :
> > dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data > UKI-LT2-IC-HEP-disk
{'Failed': {}, 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}}
3. Pass it to lcg-cp :
> > lcg-cp -v --vo gridpp > srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data > test.data
Hoping this helps,
Cheers, Raja.
On 16/03/16 15:12, Steve Lloyd wrote:
Hi, Anyone know what's wrong here. (Presumably the file got deleted but the catalogue wasn't updated). Cheers
[lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data LFN StorageElement URL
====================================================================================================================================================================== /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
[lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... Source SE type: SRMv2 Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 Source URL: srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data File size: 100000000 Source URL for copy: gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out # streams: 1 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst
Transfer took 9010 ms [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data file:t100.out Using grid catalog type: UNKNOWN Using grid catalog : prod-lfc-atlas.cern.ch VO name: gridpp Checksum type: None Trying SURL srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data ... [SE][Ls][SRM_INVALID_PATH] No such file or directory lcg_cp: No such file or directory
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
lcg-cp obviously (wretched spell checker) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote:
Hi Steve,
In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.)
Sam
On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
The best way is to use DIRAC tools directly for downloading the file. The tool here is :
>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data
This command should do all the needed steps under the hood, picking any available SE at random to get the file.
But if you want to go the lcg-cp way, note that the surl you pass to lcg-cp is incorrect and the cause of your error. To get the correct surl, you need to do the following :
1. Get the list of replicas (as you did) >> >> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data
LFN StorageElement URL
======================================================================================================================================================================= /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
2. Get the accessURL for any one of the replicas : >> >> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >> UKI-LT2-IC-HEP-disk
{'Failed': {}, 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}}
3. Pass it to lcg-cp : >> >> lcg-cp -v --vo gridpp >> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >> test.data
Hoping this helps,
Cheers, Raja.
On 16/03/16 15:12, Steve Lloyd wrote: > > Hi, > Anyone know what's wrong here. (Presumably the file got deleted but > the catalogue wasn't updated). > Cheers > > [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas > /gridpp/user/s/steve.lloyd/test100mb.data > LFN StorageElement URL > > ====================================================================================================================================================================== > /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > UKI-SCOTGRID-GLASGOW-disk > srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > > [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > file:t100.out > Using grid catalog type: UNKNOWN > Using grid catalog : prod-lfc-atlas.cern.ch > VO name: gridpp > Checksum type: None > Trying SURL > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > ... > Source SE type: SRMv2 > Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 > Source URL: > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > File size: 100000000 > Source URL for copy: > gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data > Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out > # streams: 1 > 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst > 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst > 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst > 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst > 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst > 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst > 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst > 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst > 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst > 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst > > Transfer took 9010 ms > [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp > srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > file:t100.out > Using grid catalog type: UNKNOWN > Using grid catalog : prod-lfc-atlas.cern.ch > VO name: gridpp > Checksum type: None > Trying SURL > srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > ... > [SE][Ls][SRM_INVALID_PATH] No such file or directory > lcg_cp: No such file or directory > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Prof Steve Lloyd > Head of School of Physics and Astronomy > Queen Mary University of London, Mile End Road, London E1 4NS, UK > E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Hi Steve, yes: lcg-cp srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data test100mb.data.qmul lx04:~ :~] ls -l test100mb.data.qmul -rw-r--r--. 1 dbauer res0 104857600 Mar 18 14:15 test100mb.data.qmul I've also tried Glasgow and that seems to work as well. Do you remember when you first uploaded these files ? I wonder if there was some temporary glitch/update gone wrong and they got written to the wrong directory. We haven't touched the dirac instance in a while though. Regards, Daniela On 18 March 2016 at 13:30, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
lcg-cp obviously (wretched spell checker) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote:
Hi Steve,
In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.)
Sam
On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> > wrote: > > Hi Steve, > > The best way is to use DIRAC tools directly for downloading the file. > The tool here is : > >>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data > > > This command should do all the needed steps under the hood, picking any > available SE at random to get the file. > > But if you want to go the lcg-cp way, note that the surl you pass to > lcg-cp is incorrect and the cause of your error. To get the correct surl, > you need to do the following : > > 1. Get the list of replicas (as you did) >>> >>> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data > > LFN StorageElement URL > > ======================================================================================================================================================================= > /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk > srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > UKI-SCOTGRID-GLASGOW-disk > srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > UKI-LT2-QMUL2-disk > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > > > 2. Get the accessURL for any one of the replicas : >>> >>> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >>> UKI-LT2-IC-HEP-disk > > {'Failed': {}, > 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': > 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} > > 3. Pass it to lcg-cp : >>> >>> lcg-cp -v --vo gridpp >>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>> test.data > > > > Hoping this helps, > > Cheers, > Raja. > > On 16/03/16 15:12, Steve Lloyd wrote: >> >> Hi, >> Anyone know what's wrong here. (Presumably the file got deleted but >> the catalogue wasn't updated). >> Cheers >> >> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >> /gridpp/user/s/steve.lloyd/test100mb.data >> LFN StorageElement URL >> >> ====================================================================================================================================================================== >> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> UKI-SCOTGRID-GLASGOW-disk >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> >> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> file:t100.out >> Using grid catalog type: UNKNOWN >> Using grid catalog : prod-lfc-atlas.cern.ch >> VO name: gridpp >> Checksum type: None >> Trying SURL >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> ... >> Source SE type: SRMv2 >> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >> Source URL: >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> File size: 100000000 >> Source URL for copy: >> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >> # streams: 1 >> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >> >> Transfer took 9010 ms >> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> file:t100.out >> Using grid catalog type: UNKNOWN >> Using grid catalog : prod-lfc-atlas.cern.ch >> VO name: gridpp >> Checksum type: None >> Trying SURL >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> ... >> [SE][Ls][SRM_INVALID_PATH] No such file or directory >> lcg_cp: No such file or directory >> >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Prof Steve Lloyd >> Head of School of Physics and Astronomy >> Queen Mary University of London, Mile End Road, London E1 4NS, UK >> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
It was some time ago - I'm going to upload some more so I'll see what happens with them. By the way I thought I understood Python till I saw this (in DataManager.py): # # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo ) Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote:
Hi Steve,
In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.)
Sam
On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
The best way is to use DIRAC tools directly for downloading the file. The tool here is :
>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data
This command should do all the needed steps under the hood, picking any available SE at random to get the file.
But if you want to go the lcg-cp way, note that the surl you pass to lcg-cp is incorrect and the cause of your error. To get the correct surl, you need to do the following :
1. Get the list of replicas (as you did) >> >> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data
LFN StorageElement URL
======================================================================================================================================================================= /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-SCOTGRID-GLASGOW-disk srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data
2. Get the accessURL for any one of the replicas : >> >> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >> UKI-LT2-IC-HEP-disk
{'Failed': {}, 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}}
3. Pass it to lcg-cp : >> >> lcg-cp -v --vo gridpp >> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >> test.data
Hoping this helps,
Cheers, Raja.
On 16/03/16 15:12, Steve Lloyd wrote: > > Hi, > Anyone know what's wrong here. (Presumably the file got deleted but > the catalogue wasn't updated). > Cheers > > [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas > /gridpp/user/s/steve.lloyd/test100mb.data > LFN StorageElement URL > > ====================================================================================================================================================================== > /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > UKI-SCOTGRID-GLASGOW-disk > srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > > [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > file:t100.out > Using grid catalog type: UNKNOWN > Using grid catalog : prod-lfc-atlas.cern.ch > VO name: gridpp > Checksum type: None > Trying SURL > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > ... > Source SE type: SRMv2 > Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 > Source URL: > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > File size: 100000000 > Source URL for copy: > gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data > Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out > # streams: 1 > 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst > 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst > 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst > 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst > 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst > 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst > 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst > 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst > 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst > 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst > > Transfer took 9010 ms > [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp > srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > file:t100.out > Using grid catalog type: UNKNOWN > Using grid catalog : prod-lfc-atlas.cern.ch > VO name: gridpp > Checksum type: None > Trying SURL > srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > ... > [SE][Ls][SRM_INVALID_PATH] No such file or directory > lcg_cp: No such file or directory > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Prof Steve Lloyd > Head of School of Physics and Astronomy > Queen Mary University of London, Mile End Road, London E1 4NS, UK > E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Now, when I stand up at the gridpp meeting and complain about the difficulties of installing/administering/debugging/fixing dirac, at least you know what I am talking about :-D Regards, Daniela On 18 March 2016 at 14:51, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
It was some time ago - I'm going to upload some more so I'll see what happens with them.
By the way I thought I understood Python till I saw this (in DataManager.py):
# # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo )
Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote:
Hi Steve,
In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.)
Sam
On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> > wrote: > > Hi Steve, > > The best way is to use DIRAC tools directly for downloading the file. > The tool here is : > >>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data > > > This command should do all the needed steps under the hood, picking any > available SE at random to get the file. > > But if you want to go the lcg-cp way, note that the surl you pass to > lcg-cp is incorrect and the cause of your error. To get the correct surl, > you need to do the following : > > 1. Get the list of replicas (as you did) >>> >>> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data > > LFN StorageElement URL > > ======================================================================================================================================================================= > /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk > srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > UKI-SCOTGRID-GLASGOW-disk > srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > UKI-LT2-QMUL2-disk > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > > > 2. Get the accessURL for any one of the replicas : >>> >>> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >>> UKI-LT2-IC-HEP-disk > > {'Failed': {}, > 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': > 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} > > 3. Pass it to lcg-cp : >>> >>> lcg-cp -v --vo gridpp >>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>> test.data > > > > Hoping this helps, > > Cheers, > Raja. > > On 16/03/16 15:12, Steve Lloyd wrote: >> >> Hi, >> Anyone know what's wrong here. (Presumably the file got deleted but >> the catalogue wasn't updated). >> Cheers >> >> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >> /gridpp/user/s/steve.lloyd/test100mb.data >> LFN StorageElement URL >> >> ====================================================================================================================================================================== >> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> UKI-SCOTGRID-GLASGOW-disk >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> >> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> file:t100.out >> Using grid catalog type: UNKNOWN >> Using grid catalog : prod-lfc-atlas.cern.ch >> VO name: gridpp >> Checksum type: None >> Trying SURL >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> ... >> Source SE type: SRMv2 >> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >> Source URL: >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> File size: 100000000 >> Source URL for copy: >> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >> # streams: 1 >> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >> >> Transfer took 9010 ms >> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> file:t100.out >> Using grid catalog type: UNKNOWN >> Using grid catalog : prod-lfc-atlas.cern.ch >> VO name: gridpp >> Checksum type: None >> Trying SURL >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> ... >> [SE][Ls][SRM_INVALID_PATH] No such file or directory >> lcg_cp: No such file or directory >> >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Prof Steve Lloyd >> Head of School of Physics and Astronomy >> Queen Mary University of London, Mile End Road, London E1 4NS, UK >> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
Indeed! +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 14:56, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Now, when I stand up at the gridpp meeting and complain about the difficulties of installing/administering/debugging/fixing dirac, at least you know what I am talking about :-D
Regards, Daniela
On 18 March 2016 at 14:51, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
It was some time ago - I'm going to upload some more so I'll see what happens with them.
By the way I thought I understood Python till I saw this (in DataManager.py):
# # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo )
Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote: > > Hi Steve, > > In general, I agree with Raja, though: not letting the DIRAC tools handle > files that are managed via DIRAC causes problems (like the one you just > encountered), as the DIRAC namespace is not the SE namespace in general > (just as with LFCs). > (In your case, Raja's method to retrieve SURLs is the best one.) > > Sam > > On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: > Hi Raja, > Thanks - at QMUL the two surls are the same (apart from a //) so it > worked whereas at Glasgow they are different so it didn't. It was confusing > that it worked at QMUL but not at Glasgow. I want to do this in a running > job to get a particular replica to measure the bandwidth which is why I'm > not using the Dirac tools. > Cheers > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Prof Steve Lloyd > Head of School of Physics and Astronomy > Queen Mary University of London, Mile End Road, London E1 4NS, UK > E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> >> wrote: >> >> Hi Steve, >> >> The best way is to use DIRAC tools directly for downloading the file. >> The tool here is : >> >>>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data >> >> >> This command should do all the needed steps under the hood, picking any >> available SE at random to get the file. >> >> But if you want to go the lcg-cp way, note that the surl you pass to >> lcg-cp is incorrect and the cause of your error. To get the correct surl, >> you need to do the following : >> >> 1. Get the list of replicas (as you did) >>>> >>>> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data >> >> LFN StorageElement URL >> >> ======================================================================================================================================================================= >> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk >> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> UKI-SCOTGRID-GLASGOW-disk >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> UKI-LT2-QMUL2-disk >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> >> >> 2. Get the accessURL for any one of the replicas : >>>> >>>> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >>>> UKI-LT2-IC-HEP-disk >> >> {'Failed': {}, >> 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': >> 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} >> >> 3. Pass it to lcg-cp : >>>> >>>> lcg-cp -v --vo gridpp >>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>>> test.data >> >> >> >> Hoping this helps, >> >> Cheers, >> Raja. >> >> On 16/03/16 15:12, Steve Lloyd wrote: >>> >>> Hi, >>> Anyone know what's wrong here. (Presumably the file got deleted but >>> the catalogue wasn't updated). >>> Cheers >>> >>> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >>> /gridpp/user/s/steve.lloyd/test100mb.data >>> LFN StorageElement URL >>> >>> ====================================================================================================================================================================== >>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> UKI-SCOTGRID-GLASGOW-disk >>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> >>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> file:t100.out >>> Using grid catalog type: UNKNOWN >>> Using grid catalog : prod-lfc-atlas.cern.ch >>> VO name: gridpp >>> Checksum type: None >>> Trying SURL >>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> ... >>> Source SE type: SRMv2 >>> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >>> Source URL: >>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> File size: 100000000 >>> Source URL for copy: >>> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >>> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >>> # streams: 1 >>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>> >>> Transfer took 9010 ms >>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> file:t100.out >>> Using grid catalog type: UNKNOWN >>> Using grid catalog : prod-lfc-atlas.cern.ch >>> VO name: gridpp >>> Checksum type: None >>> Trying SURL >>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> ... >>> [SE][Ls][SRM_INVALID_PATH] No such file or directory >>> lcg_cp: No such file or directory >>> >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Prof Steve Lloyd >>> Head of School of Physics and Astronomy >>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>> >> >> -- >> _______________________________________________ >> Gridpp-Dirac-Users mailing list >> Gridpp-Dirac-Users@imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users > > > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
On the brighter side, thank goodness it is at least written in a language and a manner which can be reasonably easily hacked to do our bidding. Cheers, Raja. On 18/03/16 15:05, Steve Lloyd wrote:
Indeed! +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 14:56, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Now, when I stand up at the gridpp meeting and complain about the difficulties of installing/administering/debugging/fixing dirac, at least you know what I am talking about :-D
Regards, Daniela
On 18 March 2016 at 14:51, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
It was some time ago - I'm going to upload some more so I'll see what happens with them.
By the way I thought I understood Python till I saw this (in DataManager.py):
# # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo )
Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote: > > Hi, > Unfortunately it appears that Raja's method does not work at QMUL. At > QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at > Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls > in the first place?) Is there a Dirac tool to retrieve a particular replica? > Cheers > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Prof Steve Lloyd > Head of School of Physics and Astronomy > Queen Mary University of London, Mile End Road, London E1 4NS, UK > E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote: >> >> Hi Steve, >> >> In general, I agree with Raja, though: not letting the DIRAC tools handle >> files that are managed via DIRAC causes problems (like the one you just >> encountered), as the DIRAC namespace is not the SE namespace in general >> (just as with LFCs). >> (In your case, Raja's method to retrieve SURLs is the best one.) >> >> Sam >> >> On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: >> Hi Raja, >> Thanks - at QMUL the two surls are the same (apart from a //) so it >> worked whereas at Glasgow they are different so it didn't. It was confusing >> that it worked at QMUL but not at Glasgow. I want to do this in a running >> job to get a particular replica to measure the bandwidth which is why I'm >> not using the Dirac tools. >> Cheers >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Prof Steve Lloyd >> Head of School of Physics and Astronomy >> Queen Mary University of London, Mile End Road, London E1 4NS, UK >> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> >>> wrote: >>> >>> Hi Steve, >>> >>> The best way is to use DIRAC tools directly for downloading the file. >>> The tool here is : >>> >>>>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data >>> >>> >>> This command should do all the needed steps under the hood, picking any >>> available SE at random to get the file. >>> >>> But if you want to go the lcg-cp way, note that the surl you pass to >>> lcg-cp is incorrect and the cause of your error. To get the correct surl, >>> you need to do the following : >>> >>> 1. Get the list of replicas (as you did) >>>>> >>>>> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data >>> >>> LFN StorageElement URL >>> >>> ======================================================================================================================================================================= >>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk >>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> UKI-SCOTGRID-GLASGOW-disk >>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> UKI-LT2-QMUL2-disk >>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> >>> >>> 2. Get the accessURL for any one of the replicas : >>>>> >>>>> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >>>>> UKI-LT2-IC-HEP-disk >>> >>> {'Failed': {}, >>> 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': >>> 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} >>> >>> 3. Pass it to lcg-cp : >>>>> >>>>> lcg-cp -v --vo gridpp >>>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>>>> test.data >>> >>> >>> >>> Hoping this helps, >>> >>> Cheers, >>> Raja. >>> >>> On 16/03/16 15:12, Steve Lloyd wrote: >>>> >>>> Hi, >>>> Anyone know what's wrong here. (Presumably the file got deleted but >>>> the catalogue wasn't updated). >>>> Cheers >>>> >>>> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >>>> /gridpp/user/s/steve.lloyd/test100mb.data >>>> LFN StorageElement URL >>>> >>>> ====================================================================================================================================================================== >>>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> UKI-SCOTGRID-GLASGOW-disk >>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> >>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> file:t100.out >>>> Using grid catalog type: UNKNOWN >>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>> VO name: gridpp >>>> Checksum type: None >>>> Trying SURL >>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> ... >>>> Source SE type: SRMv2 >>>> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >>>> Source URL: >>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> File size: 100000000 >>>> Source URL for copy: >>>> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >>>> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >>>> # streams: 1 >>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>> >>>> Transfer took 9010 ms >>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> file:t100.out >>>> Using grid catalog type: UNKNOWN >>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>> VO name: gridpp >>>> Checksum type: None >>>> Trying SURL >>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> ... >>>> [SE][Ls][SRM_INVALID_PATH] No such file or directory >>>> lcg_cp: No such file or directory >>>> >>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Prof Steve Lloyd >>>> Head of School of Physics and Astronomy >>>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >>>> >>> >>> -- >>> _______________________________________________ >>> Gridpp-Dirac-Users mailing list >>> Gridpp-Dirac-Users@imperial.ac.uk >>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >> >> >> >> -- >> _______________________________________________ >> Gridpp-Dirac-Users mailing list >> Gridpp-Dirac-Users@imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >> -- >> _______________________________________________ >> Gridpp-Dirac-Users mailing list >> Gridpp-Dirac-Users@imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users > > >
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Hi Steve, Which line is it? I did not find such an obviously strange piece of code. If you can point it out to me, I will be happy to feed it back to the DIRAC developers. Cheers, Raja. On 18/03/16 14:51, Steve Lloyd wrote:
It was some time ago - I'm going to upload some more so I'll see what happens with them.
By the way I thought I understood Python till I saw this (in DataManager.py):
# # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo )
Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote:
Hi Steve,
In general, I agree with Raja, though: not letting the DIRAC tools handle files that are managed via DIRAC causes problems (like the one you just encountered), as the DIRAC namespace is not the SE namespace in general (just as with LFCs). (In your case, Raja's method to retrieve SURLs is the best one.)
Sam
On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: Hi Raja, Thanks - at QMUL the two surls are the same (apart from a //) so it worked whereas at Glasgow they are different so it didn't. It was confusing that it worked at QMUL but not at Glasgow. I want to do this in a running job to get a particular replica to measure the bandwidth which is why I'm not using the Dirac tools. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> > wrote: > > Hi Steve, > > The best way is to use DIRAC tools directly for downloading the file. > The tool here is : > >>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data > > > This command should do all the needed steps under the hood, picking any > available SE at random to get the file. > > But if you want to go the lcg-cp way, note that the surl you pass to > lcg-cp is incorrect and the cause of your error. To get the correct surl, > you need to do the following : > > 1. Get the list of replicas (as you did) >>> >>> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data > > LFN StorageElement URL > > ======================================================================================================================================================================= > /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk > srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > UKI-SCOTGRID-GLASGOW-disk > srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > UKI-LT2-QMUL2-disk > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data > > > 2. Get the accessURL for any one of the replicas : >>> >>> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >>> UKI-LT2-IC-HEP-disk > > {'Failed': {}, > 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': > 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} > > 3. Pass it to lcg-cp : >>> >>> lcg-cp -v --vo gridpp >>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>> test.data > > > > Hoping this helps, > > Cheers, > Raja. > > On 16/03/16 15:12, Steve Lloyd wrote: >> >> Hi, >> Anyone know what's wrong here. (Presumably the file got deleted but >> the catalogue wasn't updated). >> Cheers >> >> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >> /gridpp/user/s/steve.lloyd/test100mb.data >> LFN StorageElement URL >> >> ====================================================================================================================================================================== >> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> UKI-SCOTGRID-GLASGOW-disk >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> >> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> file:t100.out >> Using grid catalog type: UNKNOWN >> Using grid catalog : prod-lfc-atlas.cern.ch >> VO name: gridpp >> Checksum type: None >> Trying SURL >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> ... >> Source SE type: SRMv2 >> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >> Source URL: >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> File size: 100000000 >> Source URL for copy: >> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >> # streams: 1 >> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >> >> Transfer took 9010 ms >> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> file:t100.out >> Using grid catalog type: UNKNOWN >> Using grid catalog : prod-lfc-atlas.cern.ch >> VO name: gridpp >> Checksum type: None >> Trying SURL >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> ... >> [SE][Ls][SRM_INVALID_PATH] No such file or directory >> lcg_cp: No such file or directory >> >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Prof Steve Lloyd >> Head of School of Physics and Astronomy >> Queen Mary University of London, Mile End Road, London E1 4NS, UK >> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Hi Raja, It's DIRAC/DataManagementSystem/Client/DataManager.py line 1640. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 15:21, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Which line is it? I did not find such an obviously strange piece of code. If you can point it out to me, I will be happy to feed it back to the DIRAC developers.
Cheers, Raja.
On 18/03/16 14:51, Steve Lloyd wrote:
It was some time ago - I'm going to upload some more so I'll see what happens with them.
By the way I thought I understood Python till I saw this (in DataManager.py):
# # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo )
Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote:
Hi, Unfortunately it appears that Raja's method does not work at QMUL. At QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls in the first place?) Is there a Dirac tool to retrieve a particular replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote: > > Hi Steve, > > In general, I agree with Raja, though: not letting the DIRAC tools handle > files that are managed via DIRAC causes problems (like the one you just > encountered), as the DIRAC namespace is not the SE namespace in general > (just as with LFCs). > (In your case, Raja's method to retrieve SURLs is the best one.) > > Sam > > On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: > Hi Raja, > Thanks - at QMUL the two surls are the same (apart from a //) so it > worked whereas at Glasgow they are different so it didn't. It was confusing > that it worked at QMUL but not at Glasgow. I want to do this in a running > job to get a particular replica to measure the bandwidth which is why I'm > not using the Dirac tools. > Cheers > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Prof Steve Lloyd > Head of School of Physics and Astronomy > Queen Mary University of London, Mile End Road, London E1 4NS, UK > E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> >> wrote: >> >> Hi Steve, >> >> The best way is to use DIRAC tools directly for downloading the file. >> The tool here is : >> >>>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data >> >> >> This command should do all the needed steps under the hood, picking any >> available SE at random to get the file. >> >> But if you want to go the lcg-cp way, note that the surl you pass to >> lcg-cp is incorrect and the cause of your error. To get the correct surl, >> you need to do the following : >> >> 1. Get the list of replicas (as you did) >>>> >>>> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data >> >> LFN StorageElement URL >> >> ======================================================================================================================================================================= >> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk >> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> UKI-SCOTGRID-GLASGOW-disk >> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> UKI-LT2-QMUL2-disk >> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >> >> >> 2. Get the accessURL for any one of the replicas : >>>> >>>> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >>>> UKI-LT2-IC-HEP-disk >> >> {'Failed': {}, >> 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': >> 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} >> >> 3. Pass it to lcg-cp : >>>> >>>> lcg-cp -v --vo gridpp >>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>>> test.data >> >> >> >> Hoping this helps, >> >> Cheers, >> Raja. >> >> On 16/03/16 15:12, Steve Lloyd wrote: >>> >>> Hi, >>> Anyone know what's wrong here. (Presumably the file got deleted but >>> the catalogue wasn't updated). >>> Cheers >>> >>> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >>> /gridpp/user/s/steve.lloyd/test100mb.data >>> LFN StorageElement URL >>> >>> ====================================================================================================================================================================== >>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> UKI-SCOTGRID-GLASGOW-disk >>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> >>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> file:t100.out >>> Using grid catalog type: UNKNOWN >>> Using grid catalog : prod-lfc-atlas.cern.ch >>> VO name: gridpp >>> Checksum type: None >>> Trying SURL >>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> ... >>> Source SE type: SRMv2 >>> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >>> Source URL: >>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> File size: 100000000 >>> Source URL for copy: >>> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >>> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >>> # streams: 1 >>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>> >>> Transfer took 9010 ms >>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> file:t100.out >>> Using grid catalog type: UNKNOWN >>> Using grid catalog : prod-lfc-atlas.cern.ch >>> VO name: gridpp >>> Checksum type: None >>> Trying SURL >>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> ... >>> [SE][Ls][SRM_INVALID_PATH] No such file or directory >>> lcg_cp: No such file or directory >>> >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Prof Steve Lloyd >>> Head of School of Physics and Astronomy >>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>> >> >> -- >> _______________________________________________ >> Gridpp-Dirac-Users mailing list >> Gridpp-Dirac-Users@imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users > > > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Hi Steve, Thanks - it is funny, I do not find it in the github repository of the file at https://github.com/DIRACGrid/DIRAC/blob/v6r14p19/DataManagementSystem/Client... I will ask. Cheers, Raja. On 19/03/16 16:33, Steve Lloyd wrote:
Hi Raja, It's DIRAC/DataManagementSystem/Client/DataManager.py line 1640. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 15:21, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Which line is it? I did not find such an obviously strange piece of code. If you can point it out to me, I will be happy to feed it back to the DIRAC developers.
Cheers, Raja.
On 18/03/16 14:51, Steve Lloyd wrote:
It was some time ago - I'm going to upload some more so I'll see what happens with them.
By the way I thought I understood Python till I saw this (in DataManager.py):
# # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo )
Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Interesting. Unfortunately, I think that the attitude taken currently in the DIRAC DMS system is that the users do not care where they get their data from. So, the DIRAC tool does not currently provide any option to choose a specific SE.
In previous such cases, I have taken the path of hacking into DIRAC to force it to do exactly what I want. In this case, you may want to look at the function "getFile" which is defined in for example
/cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py
It looks easy enough to implement on a private script ... :)
Cheers, Raja.
On 17/03/16 10:14, Steve Lloyd wrote: > > Hi, > Unfortunately it appears that Raja's method does not work at QMUL. At > QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at > Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls > in the first place?) Is there a Dirac tool to retrieve a particular replica? > Cheers > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Prof Steve Lloyd > Head of School of Physics and Astronomy > Queen Mary University of London, Mile End Road, London E1 4NS, UK > E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote: >> >> Hi Steve, >> >> In general, I agree with Raja, though: not letting the DIRAC tools handle >> files that are managed via DIRAC causes problems (like the one you just >> encountered), as the DIRAC namespace is not the SE namespace in general >> (just as with LFCs). >> (In your case, Raja's method to retrieve SURLs is the best one.) >> >> Sam >> >> On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: >> Hi Raja, >> Thanks - at QMUL the two surls are the same (apart from a //) so it >> worked whereas at Glasgow they are different so it didn't. It was confusing >> that it worked at QMUL but not at Glasgow. I want to do this in a running >> job to get a particular replica to measure the bandwidth which is why I'm >> not using the Dirac tools. >> Cheers >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Prof Steve Lloyd >> Head of School of Physics and Astronomy >> Queen Mary University of London, Mile End Road, London E1 4NS, UK >> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> >>> wrote: >>> >>> Hi Steve, >>> >>> The best way is to use DIRAC tools directly for downloading the file. >>> The tool here is : >>> >>>>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data >>> >>> >>> This command should do all the needed steps under the hood, picking any >>> available SE at random to get the file. >>> >>> But if you want to go the lcg-cp way, note that the surl you pass to >>> lcg-cp is incorrect and the cause of your error. To get the correct surl, >>> you need to do the following : >>> >>> 1. Get the list of replicas (as you did) >>>>> >>>>> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data >>> >>> LFN StorageElement URL >>> >>> ======================================================================================================================================================================= >>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk >>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> UKI-SCOTGRID-GLASGOW-disk >>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> UKI-LT2-QMUL2-disk >>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>> >>> >>> 2. Get the accessURL for any one of the replicas : >>>>> >>>>> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >>>>> UKI-LT2-IC-HEP-disk >>> >>> {'Failed': {}, >>> 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': >>> 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} >>> >>> 3. Pass it to lcg-cp : >>>>> >>>>> lcg-cp -v --vo gridpp >>>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>>>> test.data >>> >>> >>> >>> Hoping this helps, >>> >>> Cheers, >>> Raja. >>> >>> On 16/03/16 15:12, Steve Lloyd wrote: >>>> >>>> Hi, >>>> Anyone know what's wrong here. (Presumably the file got deleted but >>>> the catalogue wasn't updated). >>>> Cheers >>>> >>>> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >>>> /gridpp/user/s/steve.lloyd/test100mb.data >>>> LFN StorageElement URL >>>> >>>> ====================================================================================================================================================================== >>>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> UKI-SCOTGRID-GLASGOW-disk >>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> >>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> file:t100.out >>>> Using grid catalog type: UNKNOWN >>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>> VO name: gridpp >>>> Checksum type: None >>>> Trying SURL >>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> ... >>>> Source SE type: SRMv2 >>>> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >>>> Source URL: >>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> File size: 100000000 >>>> Source URL for copy: >>>> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >>>> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >>>> # streams: 1 >>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>> >>>> Transfer took 9010 ms >>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> file:t100.out >>>> Using grid catalog type: UNKNOWN >>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>> VO name: gridpp >>>> Checksum type: None >>>> Trying SURL >>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> ... >>>> [SE][Ls][SRM_INVALID_PATH] No such file or directory >>>> lcg_cp: No such file or directory >>>> >>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Prof Steve Lloyd >>>> Head of School of Physics and Astronomy >>>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >>>> >>> >>> -- >>> _______________________________________________ >>> Gridpp-Dirac-Users mailing list >>> Gridpp-Dirac-Users@imperial.ac.uk >>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >> >> >> >> -- >> _______________________________________________ >> Gridpp-Dirac-Users mailing list >> Gridpp-Dirac-Users@imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >> -- >> _______________________________________________ >> Gridpp-Dirac-Users mailing list >> Gridpp-Dirac-Users@imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users > > >
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
And asking around, it is fixed already. https://github.com/DIRACGrid/DIRAC/commit/536802ae2fbd4d891be3c5bf947a87aa3d... Cheers, Raja. On 21/03/16 05:27, Raja Nandakumar wrote:
Hi Steve,
Thanks - it is funny, I do not find it in the github repository of the file at
https://github.com/DIRACGrid/DIRAC/blob/v6r14p19/DataManagementSystem/Client...
I will ask.
Cheers, Raja.
On 19/03/16 16:33, Steve Lloyd wrote:
Hi Raja, It's DIRAC/DataManagementSystem/Client/DataManager.py line 1640. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 15:21, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Which line is it? I did not find such an obviously strange piece of code. If you can point it out to me, I will be happy to feed it back to the DIRAC developers.
Cheers, Raja.
On 18/03/16 14:51, Steve Lloyd wrote:
It was some time ago - I'm going to upload some more so I'll see what happens with them.
By the way I thought I understood Python till I saw this (in DataManager.py):
# # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo )
Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 17 Mar 2016, at 17:26, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Steve,
I can't quite reproduce your problem. I do the following: I upload a file to the Imperial SE using dirac-dms-add-file lx05:2016_Jan_18_1437 :~] dirac-dms-add-file /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/daniela.bauer/test100mb.data Successfully uploaded file to UKI-LT2-IC-HEP-disk
then I replicate it to QMUL: lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': {'register': 0.09097790718078613,
'replicate': 17.55987310409546}}}
then I ask for the accessURL: lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
and
lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}}
which are both correct and I can use it with lcg-ls (not in same terminal though): lx04:~ :~] lcg-ls srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /gridpp/gridpp/user/d/daniela.bauer/test100mb.data
lx04:~ :~] lcg-ls srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data
When you uploaded your files, did you mix and match lcg and dirac tools ?
Regards, Daniela
On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> wrote: > Hi Steve, > > Interesting. Unfortunately, I think that the attitude taken currently in the > DIRAC DMS system is that the users do not care where they get their data > from. So, the DIRAC tool does not currently provide any option to choose a > specific SE. > > In previous such cases, I have taken the path of hacking into DIRAC to force > it to do exactly what I want. In this case, you may want to look at the > function "getFile" which is defined in for example > > /cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py > > It looks easy enough to implement on a private script ... :) > > Cheers, > Raja. > > > On 17/03/16 10:14, Steve Lloyd wrote: >> >> Hi, >> Unfortunately it appears that Raja's method does not work at QMUL. At >> QMUL you have to use the surl returned by dirac-dms-lfn-replicas and at >> Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there 2 surls >> in the first place?) Is there a Dirac tool to retrieve a particular replica? >> Cheers >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Prof Steve Lloyd >> Head of School of Physics and Astronomy >> Queen Mary University of London, Mile End Road, London E1 4NS, UK >> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> wrote: >>> >>> Hi Steve, >>> >>> In general, I agree with Raja, though: not letting the DIRAC tools handle >>> files that are managed via DIRAC causes problems (like the one you just >>> encountered), as the DIRAC namespace is not the SE namespace in general >>> (just as with LFCs). >>> (In your case, Raja's method to retrieve SURLs is the best one.) >>> >>> Sam >>> >>> On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote: >>> Hi Raja, >>> Thanks - at QMUL the two surls are the same (apart from a //) so it >>> worked whereas at Glasgow they are different so it didn't. It was confusing >>> that it worked at QMUL but not at Glasgow. I want to do this in a running >>> job to get a particular replica to measure the bandwidth which is why I'm >>> not using the Dirac tools. >>> Cheers >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Prof Steve Lloyd >>> Head of School of Physics and Astronomy >>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>>> On 16 Mar 2016, at 15:32, Raja Nandakumar <raja.nandakumar@cern.ch> >>>> wrote: >>>> >>>> Hi Steve, >>>> >>>> The best way is to use DIRAC tools directly for downloading the file. >>>> The tool here is : >>>> >>>>>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data >>>> >>>> >>>> This command should do all the needed steps under the hood, picking any >>>> available SE at random to get the file. >>>> >>>> But if you want to go the lcg-cp way, note that the surl you pass to >>>> lcg-cp is incorrect and the cause of your error. To get the correct surl, >>>> you need to do the following : >>>> >>>> 1. Get the list of replicas (as you did) >>>>>> >>>>>> dirac-dms-lfn-replicas /gridpp/user/s/steve.lloyd/test100mb.data >>>> >>>> LFN StorageElement URL >>>> >>>> ======================================================================================================================================================================= >>>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk >>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> UKI-SCOTGRID-GLASGOW-disk >>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> UKI-LT2-QMUL2-disk >>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>> >>>> >>>> 2. Get the accessURL for any one of the replicas : >>>>>> >>>>>> dirac-dms-lfn-accessURL /gridpp/user/s/steve.lloyd/test100mb.data >>>>>> UKI-LT2-IC-HEP-disk >>>> >>>> {'Failed': {}, >>>> 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': >>>> 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} >>>> >>>> 3. Pass it to lcg-cp : >>>>>> >>>>>> lcg-cp -v --vo gridpp >>>>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>>>>> test.data >>>> >>>> >>>> >>>> Hoping this helps, >>>> >>>> Cheers, >>>> Raja. >>>> >>>> On 16/03/16 15:12, Steve Lloyd wrote: >>>>> >>>>> Hi, >>>>> Anyone know what's wrong here. (Presumably the file got deleted but >>>>> the catalogue wasn't updated). >>>>> Cheers >>>>> >>>>> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >>>>> /gridpp/user/s/steve.lloyd/test100mb.data >>>>> LFN StorageElement URL >>>>> >>>>> ====================================================================================================================================================================== >>>>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> UKI-SCOTGRID-GLASGOW-disk >>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> >>>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> file:t100.out >>>>> Using grid catalog type: UNKNOWN >>>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>>> VO name: gridpp >>>>> Checksum type: None >>>>> Trying SURL >>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> ... >>>>> Source SE type: SRMv2 >>>>> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >>>>> Source URL: >>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> File size: 100000000 >>>>> Source URL for copy: >>>>> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >>>>> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >>>>> # streams: 1 >>>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>>> >>>>> Transfer took 9010 ms >>>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> file:t100.out >>>>> Using grid catalog type: UNKNOWN >>>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>>> VO name: gridpp >>>>> Checksum type: None >>>>> Trying SURL >>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> ... >>>>> [SE][Ls][SRM_INVALID_PATH] No such file or directory >>>>> lcg_cp: No such file or directory >>>>> >>>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>> Prof Steve Lloyd >>>>> Head of School of Physics and Astronomy >>>>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>>>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>> >>>>> >>>> >>>> -- >>>> _______________________________________________ >>>> Gridpp-Dirac-Users mailing list >>>> Gridpp-Dirac-Users@imperial.ac.uk >>>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >>> >>> >>> >>> -- >>> _______________________________________________ >>> Gridpp-Dirac-Users mailing list >>> Gridpp-Dirac-Users@imperial.ac.uk >>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >>> -- >>> _______________________________________________ >>> Gridpp-Dirac-Users mailing list >>> Gridpp-Dirac-Users@imperial.ac.uk >>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >> >> >> > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Still this is the programming equivalent of what the NHS calls a "never event" ;-) On 21 March 2016 at 09:41, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
And asking around, it is fixed already.
https://github.com/DIRACGrid/DIRAC/commit/536802ae2fbd4d891be3c5bf947a87aa3d...
Cheers, Raja.
On 21/03/16 05:27, Raja Nandakumar wrote:
Hi Steve,
Thanks - it is funny, I do not find it in the github repository of the file at
https://github.com/DIRACGrid/DIRAC/blob/v6r14p19/DataManagementSystem/Client...
I will ask.
Cheers, Raja.
On 19/03/16 16:33, Steve Lloyd wrote:
Hi Raja, It's DIRAC/DataManagementSystem/Client/DataManager.py line 1640. Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 15:21, Raja Nandakumar <raja.nandakumar@cern.ch> wrote:
Hi Steve,
Which line is it? I did not find such an obviously strange piece of code. If you can point it out to me, I will be happy to feed it back to the DIRAC developers.
Cheers, Raja.
On 18/03/16 14:51, Steve Lloyd wrote:
It was some time ago - I'm going to upload some more so I'll see what happens with them.
By the way I thought I understood Python till I saw this (in DataManager.py):
# # call StorageElement function at least se = se = se if se else StorageElement( storageElementName, vo = self.vo )
Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 18 Mar 2016, at 13:28, Steve Lloyd <s.l.lloyd@qmul.ac.uk> wrote:
Hi Daniela, I uploaded using the dirac tools. Can you do a leg-cp on your QMUL replica? Cheers +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Prof Steve Lloyd Head of School of Physics and Astronomy Queen Mary University of London, Mile End Road, London E1 4NS, UK E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> On 17 Mar 2016, at 17:26, Daniela Bauer > <daniela.bauer.grid@googlemail.com> wrote: > > Hi Steve, > > I can't quite reproduce your problem. > I do the following: > I upload a file to the Imperial SE using dirac-dms-add-file > lx05:2016_Jan_18_1437 :~] dirac-dms-add-file > /gridpp/user/d/daniela.bauer/test100mb.data test100mb.data > UKI-LT2-IC-HEP-disk > > Uploading /gridpp/user/d/daniela.bauer/test100mb.data > Successfully uploaded file to UKI-LT2-IC-HEP-disk > > then I replicate it to QMUL: > lx05:2016_Jan_18_1437 :~] dirac-dms-replicate-lfn > /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk > {'Failed': {}, > 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': > {'register': 0.09097790718078613, > > 'replicate': 17.55987310409546}}} > > > then I ask for the accessURL: > lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL > /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-QMUL2-disk > {'Failed': {}, > 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': > > 'srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}} > > and > > lx05:2016_Jan_18_1437 :~] dirac-dms-lfn-accessURL > /gridpp/user/d/daniela.bauer/test100mb.data UKI-LT2-IC-HEP-disk > {'Failed': {}, > 'Successful': {'/gridpp/user/d/daniela.bauer/test100mb.data': > > 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data'}} > > > which are both correct and I can use it with lcg-ls (not in same > terminal though): > lx04:~ :~] lcg-ls > > srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/gridpp/gridpp/user/d/daniela.bauer/test100mb.data > /gridpp/gridpp/user/d/daniela.bauer/test100mb.data > > lx04:~ :~] lcg-ls > > srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data > > /pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/d/daniela.bauer/test100mb.data > > When you uploaded your files, did you mix and match lcg and dirac > tools ? > > Regards, > Daniela > > On 17 March 2016 at 11:03, Raja Nandakumar <raja.nandakumar@cern.ch> > wrote: >> >> Hi Steve, >> >> Interesting. Unfortunately, I think that the attitude taken >> currently in the >> DIRAC DMS system is that the users do not care where they get their >> data >> from. So, the DIRAC tool does not currently provide any option to >> choose a >> specific SE. >> >> In previous such cases, I have taken the path of hacking into DIRAC >> to force >> it to do exactly what I want. In this case, you may want to look at >> the >> function "getFile" which is defined in for example >> >> >> /cvmfs/lhcb.cern.ch/lib/lhcb/DIRAC/DIRAC_v6r14p19/DIRAC/DataManagementSystem/Client/DataManager.py >> >> It looks easy enough to implement on a private script ... :) >> >> Cheers, >> Raja. >> >> >> On 17/03/16 10:14, Steve Lloyd wrote: >>> >>> >>> Hi, >>> Unfortunately it appears that Raja's method does not work at >>> QMUL. At >>> QMUL you have to use the surl returned by dirac-dms-lfn-replicas >>> and at >>> Glasgow the one returned by dirac-dms-lfn-accessURL! (Why are there >>> 2 surls >>> in the first place?) Is there a Dirac tool to retrieve a particular >>> replica? >>> Cheers >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Prof Steve Lloyd >>> Head of School of Physics and Astronomy >>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>>> On 17 Mar 2016, at 10:00, Sam Skipsey <sskipsey@googlemail.com> >>>> wrote: >>>> >>>> Hi Steve, >>>> >>>> In general, I agree with Raja, though: not letting the DIRAC tools >>>> handle >>>> files that are managed via DIRAC causes problems (like the one you >>>> just >>>> encountered), as the DIRAC namespace is not the SE namespace in >>>> general >>>> (just as with LFCs). >>>> (In your case, Raja's method to retrieve SURLs is the best one.) >>>> >>>> Sam >>>> >>>> On Wed, 16 Mar 2016 at 17:11 Steve Lloyd <s.l.lloyd@qmul.ac.uk> >>>> wrote: >>>> Hi Raja, >>>> Thanks - at QMUL the two surls are the same (apart from a //) so >>>> it >>>> worked whereas at Glasgow they are different so it didn't. It was >>>> confusing >>>> that it worked at QMUL but not at Glasgow. I want to do this in a >>>> running >>>> job to get a particular replica to measure the bandwidth which is >>>> why I'm >>>> not using the Dirac tools. >>>> Cheers >>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Prof Steve Lloyd >>>> Head of School of Physics and Astronomy >>>> Queen Mary University of London, Mile End Road, London E1 4NS, UK >>>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >>>>> On 16 Mar 2016, at 15:32, Raja Nandakumar >>>>> <raja.nandakumar@cern.ch> >>>>> wrote: >>>>> >>>>> Hi Steve, >>>>> >>>>> The best way is to use DIRAC tools directly for downloading the >>>>> file. >>>>> The tool here is : >>>>> >>>>>>> dirac-dms-get-file /gridpp/user/s/steve.lloyd/test100mb.data >>>>> >>>>> >>>>> >>>>> This command should do all the needed steps under the hood, >>>>> picking any >>>>> available SE at random to get the file. >>>>> >>>>> But if you want to go the lcg-cp way, note that the surl you pass >>>>> to >>>>> lcg-cp is incorrect and the cause of your error. To get the >>>>> correct surl, >>>>> you need to do the following : >>>>> >>>>> 1. Get the list of replicas (as you did) >>>>>>> >>>>>>> >>>>>>> dirac-dms-lfn-replicas >>>>>>> /gridpp/user/s/steve.lloyd/test100mb.data >>>>> >>>>> >>>>> LFN StorageElement >>>>> URL >>>>> >>>>> >>>>> ======================================================================================================================================================================= >>>>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-IC-HEP-disk >>>>> >>>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> >>>>> UKI-SCOTGRID-GLASGOW-disk >>>>> >>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> UKI-LT2-QMUL2-disk >>>>> >>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>> >>>>> >>>>> 2. Get the accessURL for any one of the replicas : >>>>>>> >>>>>>> >>>>>>> dirac-dms-lfn-accessURL >>>>>>> /gridpp/user/s/steve.lloyd/test100mb.data >>>>>>> UKI-LT2-IC-HEP-disk >>>>> >>>>> >>>>> {'Failed': {}, >>>>> 'Successful': {'/gridpp/user/s/steve.lloyd/test100mb.data': >>>>> >>>>> 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data'}} >>>>> >>>>> 3. Pass it to lcg-cp : >>>>>>> >>>>>>> >>>>>>> lcg-cp -v --vo gridpp >>>>>>> >>>>>>> srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/gridpp/gridpp/user/s/steve.lloyd/test100mb.data >>>>>>> test.data >>>>> >>>>> >>>>> >>>>> >>>>> Hoping this helps, >>>>> >>>>> Cheers, >>>>> Raja. >>>>> >>>>> On 16/03/16 15:12, Steve Lloyd wrote: >>>>>> >>>>>> >>>>>> Hi, >>>>>> Anyone know what's wrong here. (Presumably the file got >>>>>> deleted but >>>>>> the catalogue wasn't updated). >>>>>> Cheers >>>>>> >>>>>> [lloyd@atlastest gridtests]$ dirac-dms-lfn-replicas >>>>>> /gridpp/user/s/steve.lloyd/test100mb.data >>>>>> LFN StorageElement >>>>>> URL >>>>>> >>>>>> >>>>>> ====================================================================================================================================================================== >>>>>> /gridpp/user/s/steve.lloyd/test100mb.data UKI-LT2-QMUL2-disk >>>>>> >>>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> >>>>>> UKI-SCOTGRID-GLASGOW-disk >>>>>> >>>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> >>>>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>>>> >>>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> file:t100.out >>>>>> Using grid catalog type: UNKNOWN >>>>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>>>> VO name: gridpp >>>>>> Checksum type: None >>>>>> Trying SURL >>>>>> >>>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> ... >>>>>> Source SE type: SRMv2 >>>>>> Source SRM Request Token: 8f607cdd-0c39-4b3f-abcb-4f5b36dd1df8 >>>>>> Source URL: >>>>>> >>>>>> srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> File size: 100000000 >>>>>> Source URL for copy: >>>>>> >>>>>> gsiftp://se03.esc.qmul.ac.uk:2811//mnt/lustre_2/storm_3/gridpp/user/s/steve.lloyd/test100mb.data >>>>>> Destination URL: file:/home/lloyd/gridpp/gridtests/t100.out >>>>>> # streams: 1 >>>>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst >>>>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>>>> 1048576 bytes 511.89 KB/sec avg 511.89 KB/sec inst >>>>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>>>> 5242880 bytes 1279.65 KB/sec avg 2047.30 KB/sec inst >>>>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>>>> 14680064 bytes 2389.23 KB/sec avg 4609.92 KB/sec inst >>>>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>>>> 90177536 bytes 11007.91 KB/sec avg 36867.80 KB/sec inst >>>>>> >>>>>> Transfer took 9010 ms >>>>>> [lloyd@atlastest gridtests]$ lcg-cp -v --vo gridpp >>>>>> >>>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> file:t100.out >>>>>> Using grid catalog type: UNKNOWN >>>>>> Using grid catalog : prod-lfc-atlas.cern.ch >>>>>> VO name: gridpp >>>>>> Checksum type: None >>>>>> Trying SURL >>>>>> >>>>>> srm://svr018.gla.scotgrid.ac.uk:8446/srm/managerv2?SFN=//gridpp/user/s/steve.lloyd/test100mb.data >>>>>> ... >>>>>> [SE][Ls][SRM_INVALID_PATH] No such file or directory >>>>>> lcg_cp: No such file or directory >>>>>> >>>>>> >>>>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> Prof Steve Lloyd >>>>>> Head of School of Physics and Astronomy >>>>>> Queen Mary University of London, Mile End Road, London E1 4NS, >>>>>> UK >>>>>> E-mail: s.l.lloyd@qmul.ac.uk Phone: +44-(0)207-882-6967 >>>>>> >>>>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> >>>>>> >>>>> >>>>> -- >>>>> _______________________________________________ >>>>> Gridpp-Dirac-Users mailing list >>>>> Gridpp-Dirac-Users@imperial.ac.uk >>>>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >>>> >>>> >>>> >>>> >>>> -- >>>> _______________________________________________ >>>> Gridpp-Dirac-Users mailing list >>>> Gridpp-Dirac-Users@imperial.ac.uk >>>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >>>> -- >>>> _______________________________________________ >>>> Gridpp-Dirac-Users mailing list >>>> Gridpp-Dirac-Users@imperial.ac.uk >>>> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users >>> >>> >>> >>> >> >> -- >> _______________________________________________ >> Gridpp-Dirac-Users mailing list >> Gridpp-Dirac-Users@imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users > > > > > -- > Sent from the pit of despair > > ----------------------------------------------------------- > daniela.bauer@imperial.ac.uk > HEP Group/Physics Dep > Imperial College > London, SW7 2BW > Tel: +44-(0)20-75947810 > http://www.hep.ph.ic.ac.uk/~dbauer/ > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users@imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
participants (4)
-
Daniela Bauer
-
Raja Nandakumar
-
Sam Skipsey
-
Steve Lloyd