*******************
This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender.
If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address.
*******************
Hi Everyone
As far as I can tell, DIRAC does not like me uploading files that end in
.cfg to the DFC.
Below you can see my attempts at uploading a file with .cfg and how the
problem is fixed if I move it to a .txt file.
Both local and lfn have what.cfg
- nothing happens
- get no complaint, but if I check, the file has not uploaded
king@heppc105:~ $ dirac-dms-add-file
/t2k.org/user/k/sophie.king/what/what*.cfg* ./what*.cfg* CA-SFU-T21-disk
Try again with debug
- no error, just seems to stop, and still no upload
king@heppc105:~ $ dirac-dms-add-file
/t2k.org/user/k/sophie.king/what/what*.cfg* ./what*.cfg* CA-SFU-T21-disk
-ddd
2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: dirac.cfg
should be at /.data/king/grid/dirac/dirac_ui/etc/dirac.cfg
2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: CFG merged
2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: Updating
configuration internals
2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: Updating
configuration internals
Parsing command line
Loading file /t2k.org/user/k/sophie.king/what/what.cfg
Could not load file /t2k.org/user/k/sophie.king/what/what.cfg: Can't
load a cfg file '/t2k.org/user/k/sophie.king/what/what.cfg'
Loading file ./what.cfg
CFG merged
Updating configuration internals
Trying to load Resources.LogBackends.StdoutBackend
Trying to load DIRAC.Resources.LogBackends.StdoutBackend
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Refreshing configuration...
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Refreshing from list
['dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server',
'dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server',
'dips://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server']
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Randomized server list is
dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server,
dips://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server,
dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Already given a valid url
dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to refresh from
dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to connect to:
dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Using proxy /tmp/x509up_u32226
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: CAs
location is /etc/grid-security/certificates
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Loaded 92 CAs [184 CRLs]
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Connection timeout set to: 1
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Authenticated peer
(/C=UK/O=eScience/OU=Imperial/L=Physics/CN=dirac02.grid.hep.ph.ic.ac.uk)
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New
session connecting to server at ('dirac02.grid.hep.ph.ic.ac.uk', 9135)
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Connected to: dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New
connection -> 146.179.232.11:9135
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:
Closing socket
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New
version available Updating to version 2020-04-12 18:53:18.780475...
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Updating configuration internals
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Updated to version 2020-04-12 18:53:18.780475
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to load Resources.LogBackends.StdoutBackend
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to load WebAppDIRAC.Resources.LogBackends.StdoutBackend
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to load GridPPDIRAC.Resources.LogBackends.StdoutBackend
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to load VMDIRAC.Resources.LogBackends.StdoutBackend
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to load DIRAC.Resources.LogBackends.StdoutBackend
2020-04-12 19:27:17 UTC dirac-dms-add-file/FileCatalogFactory
[140440172357376] DEBUG: Creating FileCatalog client of type FileCatalog
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to load Resources.Catalog.FileCatalogClient
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to load WebAppDIRAC.Resources.Catalog.FileCatalogClient
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to load GridPPDIRAC.Resources.Catalog.FileCatalogClient
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to load VMDIRAC.Resources.Catalog.FileCatalogClient
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG:
Trying to load DIRAC.Resources.Catalog.FileCatalogClient
2020-04-12 19:27:17 UTC dirac-dms-add-file/FileCatalogFactory
[140440172357376] DEBUG: Loaded module FileCatalogClient
Keep local file as .cfg but change lfn to .txt
- get an error
king@heppc105:~ $ dirac-dms-add-file
/t2k.org/user/k/sophie.king/what/what.txt ./what.cfg CA-SFU-T21-disk
Traceback (most recent call last):
File
"/.data/king/grid/dirac/dirac_ui/DIRAC/DataManagementSystem/scripts/dirac-dms-add-file.py",
line 71, in <module>
lfns.append( getDict( args ) )
File
"/.data/king/grid/dirac/dirac_ui/DIRAC/DataManagementSystem/scripts/dirac-dms-add-file.py",
line 52, in getDict
lfn_dict['SE'] = item_list[2]
IndexError: list index out of range
Ensure both the local file and the lfn have .txt
- Finally, this does work
king@heppc105:~ $ mv what.cfg what.txt
king@heppc105:~ $ dirac-dms-add-file
/t2k.org/user/k/sophie.king/what/what.txt ./what.txt CA-SFU-T21-disk
Uploading /t2k.org/user/k/sophie.king/what/what.txt
Successfully uploaded file to CA-SFU-T21-disk
I can reproduce this problem with multiple tests.
We can do a workaround of course, but I was wondering if this a known
thing, that DIRAC doesnt like the local or LFN file to have .cfg ? And
can it be fixed or is it the by-product of something deeper and we must
just live with it?
It would seem quite sensible/common that one may have some files on the
grid, and for bookkeeping purposes want to preserve a file that ends in
.cfg that was used as input to the software that created said files.
Cheers
Sophie