This email from soph.e.king123@gmail.com originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list to disable email stamping for this address.

Hi Everyone

As far as I can tell, DIRAC does not like me uploading files that end in .cfg to the DFC.
Below you can see my attempts at uploading a file with .cfg and how the problem is fixed if I move it to a .txt file.

Both local and lfn have what.cfg
- nothing happens
- get no complaint, but if I check, the file has not uploaded

king@heppc105:~ $ dirac-dms-add-file /t2k.org/user/k/sophie.king/what/what.cfg ./what.cfg CA-SFU-T21-disk

Try again with debug
- no error, just seems to stop, and still no upload

king@heppc105:~ $ dirac-dms-add-file /t2k.org/user/k/sophie.king/what/what.cfg ./what.cfg CA-SFU-T21-disk -ddd2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: dirac.cfg should be at /.data/king/grid/dirac/dirac_ui/etc/dirac.cfg2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: CFG merged2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: Updating configuration internals2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: Updating configuration internalsParsing command lineLoading file /t2k.org/user/k/sophie.king/what/what.cfgCould not load file /t2k.org/user/k/sophie.king/what/what.cfg: Can't load a cfg file '/t2k.org/user/k/sophie.king/what/what.cfg'Loading file ./what.cfgCFG mergedUpdating configuration internalsTrying to load Resources.LogBackends.StdoutBackendTrying to load DIRAC.Resources.LogBackends.StdoutBackend2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Refreshing configuration...2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Refreshing from list ['dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server', 'dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server', 'dips://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server']2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Randomized server list is dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server, dips://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server, dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Already given a valid url dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to refresh from dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to connect to: dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Using proxy /tmp/x509up_u322262020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: CAs location is /etc/grid-security/certificates2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Loaded 92 CAs [184 CRLs]2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Connection timeout set to: 12020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Authenticated peer (/C=UK/O=eScience/OU=Imperial/L=Physics/CN=dirac02.grid.hep.ph.ic.ac.uk)2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New session connecting to server at ('dirac02.grid.hep.ph.ic.ac.uk', 9135)2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Connected to: dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New connection -> 146.179.232.11:91352020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Closing socket2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New version available Updating to version 2020-04-12 18:53:18.780475...2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Updating configuration internals2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Updated to version 2020-04-12 18:53:18.7804752020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load Resources.LogBackends.StdoutBackend2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load WebAppDIRAC.Resources.LogBackends.StdoutBackend2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load GridPPDIRAC.Resources.LogBackends.StdoutBackend2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load VMDIRAC.Resources.LogBackends.StdoutBackend2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load DIRAC.Resources.LogBackends.StdoutBackend2020-04-12 19:27:17 UTC dirac-dms-add-file/FileCatalogFactory [140440172357376] DEBUG: Creating FileCatalog client of type FileCatalog2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load Resources.Catalog.FileCatalogClient2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load WebAppDIRAC.Resources.Catalog.FileCatalogClient2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load GridPPDIRAC.Resources.Catalog.FileCatalogClient2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load VMDIRAC.Resources.Catalog.FileCatalogClient2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load DIRAC.Resources.Catalog.FileCatalogClient2020-04-12 19:27:17 UTC dirac-dms-add-file/FileCatalogFactory [140440172357376] DEBUG: Loaded module FileCatalogClient

Keep local file as .cfg but change lfn to .txt
- get an error

king@heppc105:~ $ dirac-dms-add-file /t2k.org/user/k/sophie.king/what/what.txt ./what.cfg CA-SFU-T21-diskTraceback (most recent call last): File "/.data/king/grid/dirac/dirac_ui/DIRAC/DataManagementSystem/scripts/dirac-dms-add-file.py", line 71, in <module> lfns.append( getDict( args ) ) File "/.data/king/grid/dirac/dirac_ui/DIRAC/DataManagementSystem/scripts/dirac-dms-add-file.py", line 52, in getDict lfn_dict['SE'] = item_list[2]IndexError: list index out of range

Ensure both the local file and the lfn have .txt
- Finally, this does work

king@heppc105:~ $ mv what.cfg what.txt king@heppc105:~ $ dirac-dms-add-file /t2k.org/user/k/sophie.king/what/what.txt ./what.txt CA-SFU-T21-disk Uploading /t2k.org/user/k/sophie.king/what/what.txt Successfully uploaded file to CA-SFU-T21-disk

I can reproduce this problem with multiple tests.

We can do a workaround of course, but I was wondering if this a known thing, that DIRAC doesnt like the local or LFN file to have .cfg ? And can it be fixed or is it the by-product of something deeper and we must just live with it?

It would seem quite sensible/common that one may have some files on the grid, and for bookkeeping purposes want to preserve a file that ends in .cfg that was used as input to the software that created said files.

Cheers
Sophie