secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first
Hi I'm trying to upload and replicate a file which starts off OK: $ dirac-dms-add-file -f /gridpp/user/d/duncan.rand/testfile-20171114-1 ./testfile UKI-LT2-IC-HEP-disk Uploading /gridpp/user/d/duncan.rand/testfile-20171114-1 Successfully uploaded file to UKI-LT2-IC-HEP-disk $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-LT2-RHUL-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/duncan.rand/testfile-20171114-1': {'register': 0.03875398635864258, 'replicate': 9.515014886 but then with Sheffield, Durham, Birmingham and ECDF storage elements (where the DFC seems to be defaulting to storing the SURL as a root-based SURL) I get the error below. thanks Duncan $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-NORTHGRID-SHEF-HEP-disk 171114 15:30:06 22254 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f19741ad7c0) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-DURHAM-disk 171114 15:30:09 22265 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f9510006310) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SOUTHGRID-BHAM-HEP-disk 171114 15:30:11 22277 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f50981ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-ECDF-disk 171114 15:30:19 22288 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7fd15c1ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources.
I should point out that it worked previously and that as I said the replica at ECDF has a root:// based URL: $ dirac-dms-lfn-replicas /gridpp/user/duncan.rand/testfile-20171114 LFN StorageElement URL ================================================================ /gridpp/user/duncan.rand/testfile-20171114 UKI-LT2-RHUL-disk srm://se2.ppgrid1.rhul.ac.uk:8446/srm/managerv2?SFN=/dpm/ppgrid1.rhul.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114 $ dirac-dms-replicate-lfn /gridpp/user/duncan.rand/testfile-20171114 UKI-SCOTGRID-ECDF-disk {'Failed': {}, 'Successful': {'/gridpp/user/duncan.rand/testfile-20171114': {'register': 0.0642080307006836, 'replicate': 6.708472967147827}}} $ dirac-dms-lfn-replicas /gridpp/user/duncan.rand/testfile-20171114 LFN StorageElement URL ===================================================================== /gridpp/user/duncan.rand/testfile-20171114 UKI-SCOTGRID-ECDF-disk root://srm.glite.ecdf.ed.ac.uk//dpm/ecdf.ed.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114 UKI-LT2-RHUL-disk srm://se2.ppgrid1.rhul.ac.uk:8446/srm/managerv2?SFN=/dpm/ppgrid1.rhul.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114 Duncan On 14/11/2017 15:42, Duncan Rand wrote:
Hi
I'm trying to upload and replicate a file which starts off OK:
$ dirac-dms-add-file -f /gridpp/user/d/duncan.rand/testfile-20171114-1 ./testfile UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/duncan.rand/testfile-20171114-1 Successfully uploaded file to UKI-LT2-IC-HEP-disk $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-LT2-RHUL-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/duncan.rand/testfile-20171114-1': {'register': 0.03875398635864258,
'replicate': 9.515014886
but then with Sheffield, Durham, Birmingham and ECDF storage elements (where the DFC seems to be defaulting to storing the SURL as a root-based SURL) I get the error below.
thanks Duncan
$ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-NORTHGRID-SHEF-HEP-disk 171114 15:30:06 22254 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f19741ad7c0) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-DURHAM-disk 171114 15:30:09 22265 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f9510006310) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SOUTHGRID-BHAM-HEP-disk 171114 15:30:11 22277 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f50981ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-ECDF-disk 171114 15:30:19 22288 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7fd15c1ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources.
Hi Duncan, I think this might be a topic for the storage group. When I try and replicate your problem most of the time (UKI-SCOTGRID-ECDF-disk, UKI-SOUTHGRID-BHAM-HEP-disk, UKI-SCOTGRID-DURHAM-disk) I get the error: x01:2017_Oct_03_1309_gridpp > dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test.duncan UKI-SOUTHGRID-BHAM-HEP-disk XROOTStorage.__putSingleFile: Failed to put file on storage. [ERROR] Server responded with an error: [3010] Unable to create /dpm/ ph.bham.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan; Permission denied 3010 (replace site name as applicable). In Sheffield I get: lx01:2017_Oct_03_1309_gridpp > dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test.duncan UKI-NORTHGRID-SHEF-HEP-disk XROOTStorage.__putSingleFile: Failed to put file on storage. [ERROR] Server responded with an error: [3009] Unable to create /dpm/ shef.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan; No space left on device 3009 XROOTStorage.__putSingleFile: Failed to put file on storage. [ERROR] Server responded with an error: [3009] Unable to create /dpm/ shef.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan; No space left on device 3009 {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test.duncan': {'register': 0.0416109561920166, 'replicate': 8.558074951171875}}} So in the end it works, even though there are either some broken pool nodes or the gridpp space is (nearly) full. Elena ? I would propose to move the 'cannot create directory' issue to the storage mailing list. However in your case it looks like an authentication issue. Which version of the UI are you using and what does dirac-proxy-info say ? Regards, Daniela On 14 November 2017 at 16:03, Duncan Rand <duncan.rand@imperial.ac.uk> wrote:
I should point out that it worked previously and that as I said the replica at ECDF has a root:// based URL:
$ dirac-dms-lfn-replicas /gridpp/user/duncan.rand/testfile-20171114 LFN StorageElement URL ================================================================ /gridpp/user/duncan.rand/testfile-20171114 UKI-LT2-RHUL-disk srm:// se2.ppgrid1.rhul.ac.uk:8446/srm/managerv2?SFN=/dpm/ppg rid1.rhul.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114
$ dirac-dms-replicate-lfn /gridpp/user/duncan.rand/testfile-20171114 UKI-SCOTGRID-ECDF-disk {'Failed': {}, 'Successful': {'/gridpp/user/duncan.rand/testfile-20171114': {'register': 0.0642080307006836,
'replicate': 6.708472967147827}}} $ dirac-dms-lfn-replicas /gridpp/user/duncan.rand/testfile-20171114 LFN StorageElement URL ===================================================================== /gridpp/user/duncan.rand/testfile-20171114 UKI-SCOTGRID-ECDF-disk root:// srm.glite.ecdf.ed.ac.uk//dpm/ecdf.ed.ac.uk/home/gridp p/gridpp/user/duncan.rand/testfile-20171114 UKI-LT2-RHUL-disk srm:// se2.ppgrid1.rhul.ac.uk:8446/srm/managerv2?SFN=/dpm/ppg rid1.rhul.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114
Duncan
On 14/11/2017 15:42, Duncan Rand wrote:
Hi
I'm trying to upload and replicate a file which starts off OK:
$ dirac-dms-add-file -f /gridpp/user/d/duncan.rand/testfile-20171114-1 ./testfile UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/duncan.rand/testfile-20171114-1 Successfully uploaded file to UKI-LT2-IC-HEP-disk $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-LT2-RHUL-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/duncan.rand/testfile-20171114-1': {'register': 0.03875398635864258,
'replicate': 9.515014886
but then with Sheffield, Durham, Birmingham and ECDF storage elements (where the DFC seems to be defaulting to storing the SURL as a root-based SURL) I get the error below.
thanks Duncan
$ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-NORTHGRID-SHEF-HEP-disk 171114 15:30:06 22254 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f19741ad7c0) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-DURHAM-disk 171114 15:30:09 22265 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f9510006310) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SOUTHGRID-BHAM-HEP-disk 171114 15:30:11 22277 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f50981ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-ECDF-disk 171114 15:30:19 22288 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7fd15c1ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources.
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
Hi Daniela I installed the UI with ./dirac-install -r v6r19p4 -i 27 -g v13r0 and I get $ dirac-proxy-info subject : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=duncan rand/CN=8760856367/CN=2835270531 issuer : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=duncan rand/CN=8760856367 identity : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=duncan rand timeleft : 18:29:03 DIRAC group : gridpp_user rfc : True path : /tmp/x509up_u129484 username : duncan.rand properties : NormalUser, JobMonitor VOMS : True VOMS fqan : ['/gridpp'] By the way, I see now why root is being used, it is because for those SEs 'root' is defined as AccessProtocol.1 in the dirac configuration. Do you perhaps know why some SE's have root as their defaul and some SRM? thanks Duncan On 14/11/2017 16:40, Daniela Bauer wrote:
Hi Duncan,
I think this might be a topic for the storage group. When I try and replicate your problem most of the time (UKI-SCOTGRID-ECDF-disk, UKI-SOUTHGRID-BHAM-HEP-disk, UKI-SCOTGRID-DURHAM-disk) I get the error: x01:2017_Oct_03_1309_gridpp > dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test.duncan UKI-SOUTHGRID-BHAM-HEP-disk XROOTStorage.__putSingleFile: Failed to put file on storage. [ERROR] Server responded with an error: [3010] Unable to create /dpm/ph.bham.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan <http://ph.bham.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan>; Permission denied 3010
(replace site name as applicable).
In Sheffield I get: lx01:2017_Oct_03_1309_gridpp > dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test.duncan UKI-NORTHGRID-SHEF-HEP-disk XROOTStorage.__putSingleFile: Failed to put file on storage. [ERROR] Server responded with an error: [3009] Unable to create /dpm/shef.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan <http://shef.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan>; No space left on device 3009 XROOTStorage.__putSingleFile: Failed to put file on storage. [ERROR] Server responded with an error: [3009] Unable to create /dpm/shef.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan <http://shef.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan>; No space left on device 3009 {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test.duncan': {'register': 0.0416109561920166,
'replicate': 8.558074951171875}}} So in the end it works, even though there are either some broken pool nodes or the gridpp space is (nearly) full. Elena ?
I would propose to move the 'cannot create directory' issue to the storage mailing list. However in your case it looks like an authentication issue. Which version of the UI are you using and what does dirac-proxy-info say ?
Regards, Daniela
On 14 November 2017 at 16:03, Duncan Rand <duncan.rand@imperial.ac.uk <mailto:duncan.rand@imperial.ac.uk>> wrote:
I should point out that it worked previously and that as I said the replica at ECDF has a root:// based URL:
$ dirac-dms-lfn-replicas /gridpp/user/duncan.rand/testfile-20171114 LFN StorageElement URL ================================================================ /gridpp/user/duncan.rand/testfile-20171114 UKI-LT2-RHUL-disk srm://se2.ppgrid1.rhul.ac.uk:8446/srm/managerv2?SFN=/dpm/ppgrid1.rhul.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114 <http://se2.ppgrid1.rhul.ac.uk:8446/srm/managerv2?SFN=/dpm/ppgrid1.rhul.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114>
$ dirac-dms-replicate-lfn /gridpp/user/duncan.rand/testfile-20171114 UKI-SCOTGRID-ECDF-disk {'Failed': {}, 'Successful': {'/gridpp/user/duncan.rand/testfile-20171114': {'register': 0.0642080307006836,
'replicate': 6.708472967147827}}} $ dirac-dms-lfn-replicas /gridpp/user/duncan.rand/testfile-20171114 LFN StorageElement URL ===================================================================== /gridpp/user/duncan.rand/testfile-20171114 UKI-SCOTGRID-ECDF-disk root://srm.glite.ecdf.ed.ac.uk//dpm/ecdf.ed.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114 <http://srm.glite.ecdf.ed.ac.uk//dpm/ecdf.ed.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114> UKI-LT2-RHUL-disk srm://se2.ppgrid1.rhul.ac.uk:8446/srm/managerv2?SFN=/dpm/ppgrid1.rhul.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114 <http://se2.ppgrid1.rhul.ac.uk:8446/srm/managerv2?SFN=/dpm/ppgrid1.rhul.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114>
Duncan
On 14/11/2017 15:42, Duncan Rand wrote:
Hi
I'm trying to upload and replicate a file which starts off OK:
$ dirac-dms-add-file -f /gridpp/user/d/duncan.rand/testfile-20171114-1 ./testfile UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/duncan.rand/testfile-20171114-1 Successfully uploaded file to UKI-LT2-IC-HEP-disk $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-LT2-RHUL-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/duncan.rand/testfile-20171114-1': {'register': 0.03875398635864258,
'replicate': 9.515014886 <tel:9.515014886>
but then with Sheffield, Durham, Birmingham and ECDF storage elements (where the DFC seems to be defaulting to storing the SURL as a root-based SURL) I get the error below.
thanks Duncan
$ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-NORTHGRID-SHEF-HEP-disk 171114 15:30:06 22254 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f19741ad7c0) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-DURHAM-disk 171114 15:30:09 22265 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f9510006310) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SOUTHGRID-BHAM-HEP-disk 171114 15:30:11 22277 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f50981ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-ECDF-disk 171114 15:30:19 22288 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7fd15c1ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources.
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users>
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk <mailto:daniela.bauer@imperial.ac.uk> HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
Hi Daniela When I repeat the exercise with a fresh install of the UI I am able to replicate to Durham and Bham without issue. At Sheffield I get the same permission denied error you mention XROOTStorage.__putSingleFile: Failed to put file on storage. [ERROR] Server responded with an error: [3010] Unable to create /dpm/shef.ac.uk/home/gridpp/gridpp/user/d/duncan.rand/testfile-20171114-1; Permission denied and also at ECDF: XROOTStorage.__putSingleFile: Failed to put file on storage. [ERROR] Server responded with an error: [3010] Unable to create /dpm/ecdf.ed.ac.uk/home/gridpp/gridpp/user/d/duncan.rand/testfile-20171114-1; Permission denied which I'll take up elsewhere. thanks Duncan On 14/11/2017 17:04, Duncan Rand wrote:
Hi Daniela
I installed the UI with
./dirac-install -r v6r19p4 -i 27 -g v13r0
and I get
$ dirac-proxy-info subject : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=duncan rand/CN=8760856367/CN=2835270531 issuer : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=duncan rand/CN=8760856367 identity : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=duncan rand timeleft : 18:29:03 DIRAC group : gridpp_user rfc : True path : /tmp/x509up_u129484 username : duncan.rand properties : NormalUser, JobMonitor VOMS : True VOMS fqan : ['/gridpp']
By the way, I see now why root is being used, it is because for those SEs 'root' is defined as AccessProtocol.1 in the dirac configuration. Do you perhaps know why some SE's have root as their defaul and some SRM?
thanks Duncan
On 14/11/2017 16:40, Daniela Bauer wrote:
Hi Duncan,
I think this might be a topic for the storage group. When I try and replicate your problem most of the time (UKI-SCOTGRID-ECDF-disk, UKI-SOUTHGRID-BHAM-HEP-disk, UKI-SCOTGRID-DURHAM-disk) I get the error: x01:2017_Oct_03_1309_gridpp > dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test.duncan UKI-SOUTHGRID-BHAM-HEP-disk XROOTStorage.__putSingleFile: Failed to put file on storage. [ERROR] Server responded with an error: [3010] Unable to create /dpm/ph.bham.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan <http://ph.bham.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan>;
Permission denied 3010
(replace site name as applicable).
In Sheffield I get: lx01:2017_Oct_03_1309_gridpp > dirac-dms-replicate-lfn /gridpp/user/d/daniela.bauer/test.duncan UKI-NORTHGRID-SHEF-HEP-disk XROOTStorage.__putSingleFile: Failed to put file on storage. [ERROR] Server responded with an error: [3009] Unable to create /dpm/shef.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan <http://shef.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan>; No space left on device 3009 XROOTStorage.__putSingleFile: Failed to put file on storage. [ERROR] Server responded with an error: [3009] Unable to create /dpm/shef.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan <http://shef.ac.uk/home/gridpp/gridpp/user/d/daniela.bauer/test.duncan>; No space left on device 3009 {'Failed': {}, 'Successful': {'/gridpp/user/d/daniela.bauer/test.duncan': {'register': 0.0416109561920166,
'replicate': 8.558074951171875}}} So in the end it works, even though there are either some broken pool nodes or the gridpp space is (nearly) full. Elena ?
I would propose to move the 'cannot create directory' issue to the storage mailing list. However in your case it looks like an authentication issue. Which version of the UI are you using and what does dirac-proxy-info say ?
Regards, Daniela
On 14 November 2017 at 16:03, Duncan Rand <duncan.rand@imperial.ac.uk <mailto:duncan.rand@imperial.ac.uk>> wrote:
I should point out that it worked previously and that as I said the replica at ECDF has a root:// based URL:
$ dirac-dms-lfn-replicas /gridpp/user/duncan.rand/testfile-20171114 LFN StorageElement URL ================================================================ /gridpp/user/duncan.rand/testfile-20171114 UKI-LT2-RHUL-disk
srm://se2.ppgrid1.rhul.ac.uk:8446/srm/managerv2?SFN=/dpm/ppgrid1.rhul.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114
$ dirac-dms-replicate-lfn /gridpp/user/duncan.rand/testfile-20171114 UKI-SCOTGRID-ECDF-disk {'Failed': {}, 'Successful': {'/gridpp/user/duncan.rand/testfile-20171114': {'register': 0.0642080307006836,
'replicate': 6.708472967147827}}} $ dirac-dms-lfn-replicas /gridpp/user/duncan.rand/testfile-20171114 LFN StorageElement URL ===================================================================== /gridpp/user/duncan.rand/testfile-20171114 UKI-SCOTGRID-ECDF-disk
root://srm.glite.ecdf.ed.ac.uk//dpm/ecdf.ed.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114
UKI-LT2-RHUL-disk
srm://se2.ppgrid1.rhul.ac.uk:8446/srm/managerv2?SFN=/dpm/ppgrid1.rhul.ac.uk/home/gridpp/gridpp/user/duncan.rand/testfile-20171114
Duncan
On 14/11/2017 15:42, Duncan Rand wrote:
Hi
I'm trying to upload and replicate a file which starts off OK:
$ dirac-dms-add-file -f /gridpp/user/d/duncan.rand/testfile-20171114-1 ./testfile UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/duncan.rand/testfile-20171114-1 Successfully uploaded file to UKI-LT2-IC-HEP-disk $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-LT2-RHUL-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/duncan.rand/testfile-20171114-1': {'register': 0.03875398635864258,
'replicate': 9.515014886 <tel:9.515014886>
but then with Sheffield, Durham, Birmingham and ECDF storage elements (where the DFC seems to be defaulting to storing the SURL as a root-based SURL) I get the error below.
thanks Duncan
$ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-NORTHGRID-SHEF-HEP-disk 171114 15:30:06 22254 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f19741ad7c0) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-DURHAM-disk 171114 15:30:09 22265 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f9510006310) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SOUTHGRID-BHAM-HEP-disk 171114 15:30:11 22277 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f50981ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-ECDF-disk 171114 15:30:19 22288 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7fd15c1ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources.
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users>
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk <mailto:daniela.bauer@imperial.ac.uk> HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
Hi Duncan, Which proxy are you using to try and do this because this 'just worked' for me on ECDF. (output below) ``` 19:26:29-mini.dev|~ rcurrie@mini:->source dirac_ui/bashrc 19:26:35-mini.dev|~ rcurrie@mini:->dirac-proxy-init -g gridpp_user Generating proxy... Enter Certificate password: Added VOMS attribute /gridpp Uploading proxy for gridpp_user... Proxy generated: subject : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=rcurrie/CN=675917/CN=Robert Andrew Currie/CN=proxy/CN=proxy issuer : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=rcurrie/CN=675917/CN=Robert Andrew Currie/CN=proxy identity : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=rcurrie/CN=675917/CN=Robert Andrew Currie timeleft : 23:53:58 DIRAC group : gridpp_user rfc : False path : /tmp/x509up_u1000 username : robert.andrew.currie properties : NormalUser, JobMonitor VOMS : True VOMS fqan : ['/gridpp'] Proxies uploaded: DN | Group | Until (GMT) /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=rcurrie/CN=675917/CN=Robert Andrew Currie | gridpp_user | 2018/10/12 15:40 19:26:52-mini.dev|~ rcurrie@mini:->dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-ECDF-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/duncan.rand/testfile-20171114-1': {'register': 0, 'replicate': 0}}} ``` The gridpp space is only writable by the gridpp VO and this is using a dirac_ui which I installed on Monday on a centos7-VM, Rob On 14/11/2017 03:42 PM, Duncan Rand wrote:
Hi
I'm trying to upload and replicate a file which starts off OK:
$ dirac-dms-add-file -f /gridpp/user/d/duncan.rand/testfile-20171114-1 ./testfile UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/duncan.rand/testfile-20171114-1 Successfully uploaded file to UKI-LT2-IC-HEP-disk $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-LT2-RHUL-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/duncan.rand/testfile-20171114-1': {'register': 0.03875398635864258,
'replicate': 9.515014886
but then with Sheffield, Durham, Birmingham and ECDF storage elements (where the DFC seems to be defaulting to storing the SURL as a root-based SURL) I get the error below.
thanks Duncan
$ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-NORTHGRID-SHEF-HEP-disk 171114 15:30:06 22254 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f19741ad7c0) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-DURHAM-disk 171114 15:30:09 22265 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f9510006310) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SOUTHGRID-BHAM-HEP-disk 171114 15:30:11 22277 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f50981ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-ECDF-disk 171114 15:30:19 22288 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7fd15c1ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources.
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Hi Robbie I got it to work at ECDF once I created the appropriate directory tree by copying in a file with gfal-copy. Duncan On 14/11/17 19:30, Robert Currie wrote:
Hi Duncan,
Which proxy are you using to try and do this because this 'just worked' for me on ECDF. (output below)
``` 19:26:29-mini.dev|~ rcurrie@mini:->source dirac_ui/bashrc 19:26:35-mini.dev|~ rcurrie@mini:->dirac-proxy-init -g gridpp_user Generating proxy... Enter Certificate password: Added VOMS attribute /gridpp Uploading proxy for gridpp_user... Proxy generated: subject : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=rcurrie/CN=675917/CN=Robert Andrew Currie/CN=proxy/CN=proxy issuer : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=rcurrie/CN=675917/CN=Robert Andrew Currie/CN=proxy identity : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=rcurrie/CN=675917/CN=Robert Andrew Currie timeleft : 23:53:58 DIRAC group : gridpp_user rfc : False path : /tmp/x509up_u1000 username : robert.andrew.currie properties : NormalUser, JobMonitor VOMS : True VOMS fqan : ['/gridpp']
Proxies uploaded: DN | Group | Until (GMT) /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=rcurrie/CN=675917/CN=Robert Andrew Currie | gridpp_user | 2018/10/12 15:40 19:26:52-mini.dev|~ rcurrie@mini:->dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-ECDF-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/duncan.rand/testfile-20171114-1': {'register': 0, 'replicate': 0}}} ```
The gridpp space is only writable by the gridpp VO and this is using a dirac_ui which I installed on Monday on a centos7-VM,
Rob
On 14/11/2017 03:42 PM, Duncan Rand wrote:
Hi
I'm trying to upload and replicate a file which starts off OK:
$ dirac-dms-add-file -f /gridpp/user/d/duncan.rand/testfile-20171114-1 ./testfile UKI-LT2-IC-HEP-disk
Uploading /gridpp/user/d/duncan.rand/testfile-20171114-1 Successfully uploaded file to UKI-LT2-IC-HEP-disk $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-LT2-RHUL-disk {'Failed': {}, 'Successful': {'/gridpp/user/d/duncan.rand/testfile-20171114-1': {'register': 0.03875398635864258,
'replicate': 9.515014886
but then with Sheffield, Durham, Birmingham and ECDF storage elements (where the DFC seems to be defaulting to storing the SURL as a root-based SURL) I get the error below.
thanks Duncan
$ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-NORTHGRID-SHEF-HEP-disk 171114 15:30:06 22254 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f19741ad7c0) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-DURHAM-disk 171114 15:30:09 22265 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f9510006310) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SOUTHGRID-BHAM-HEP-disk 171114 15:30:11 22277 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7f50981ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources. $ dirac-dms-replicate-lfn /gridpp/user/d/duncan.rand/testfile-20171114-1 UKI-SCOTGRID-ECDF-disk 171114 15:30:19 22288 secgsi_GetCA: CRL entry for 'ffc3d59b:1' needs refreshing: clean the related entry cache first (0x7fd15c1ad940) XROOTStorage.__putSingleFile: Failed to put file on storage. [FATAL] Auth failed 0 ERROR Completely failed to replicate file. Failed to replicate with all sources.
participants (3)
-
Daniela Bauer
-
Duncan Rand
-
Robert Currie