Hi Sophie, there is a workaround: You can put all arguments into a file and give that to the command: lx02:2020_May_27_1434_lz_EL7 > cat uploadme /gridpp/daniela.bauer/bah/test2.cfg test.cfg UKI-LT2-IC-HEP-disk lx02:2020_May_27_1434_lz_EL7 > dirac-dms-add-file uploadme Uploading /gridpp/daniela.bauer/bah/test2.cfg Successfully uploaded file to UKI-LT2-IC-HEP-disk and similarly for the download command: lx02:2020_May_27_1434_lz_EL7 > cat getme /gridpp/daniela.bauer/bah/test2.cfg lx02:2020_May_27_1434_lz_EL7 > dirac-dms-get-file getme {'Failed': {}, 'Successful': {'/gridpp/daniela.bauer/bah/test2.cfg': '/home/hep/dbauer/check_test_dirac/2020_May_27_1434_lz_EL7/test2.cfg'}} If you are using it inside a dirac job as an input file, it should really go into the inputsandbox as InputSandbox = {"diractest.sh", "LFN:/gridpp/daniela.bauer/bah/test2.cfg"}; rather than downloading it by hand. Regards, Daniela On Tue, 16 Jun 2020 at 21:31, Daniela Bauer < daniela.bauer.grid@googlemail.com> wrote:
Hi Sophie,
%^&*(!!! You are right. Simon had a quick look and it seems to indicate that there is no easy solution to this. We'll get back to you (possible after a detour via the DIRAC developers list.)
Regards, Daniela
On Tue, 16 Jun 2020 at 19:32, Sophie King <soph.e.king123@gmail.com> wrote:
This email from soph.e.king123@gmail.com originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list <https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address.
Hi Everyone
As far as I can tell, DIRAC does not like me uploading files that end in .cfg to the DFC. Below you can see my attempts at uploading a file with .cfg and how the problem is fixed if I move it to a .txt file.
Both local and lfn have what.cfg - nothing happens - get no complaint, but if I check, the file has not uploaded
king@heppc105:~ $ dirac-dms-add-file / t2k.org/user/k/sophie.king/what/what*.cfg* ./what*.cfg* CA-SFU-T21-disk
Try again with debug - no error, just seems to stop, and still no upload
king@heppc105:~ $ dirac-dms-add-file / t2k.org/user/k/sophie.king/what/what*.cfg* ./what*.cfg* CA-SFU-T21-disk -ddd
2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: dirac.cfg should be at /.data/king/grid/dirac/dirac_ui/etc/dirac.cfg 2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: CFG merged 2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: Updating configuration internals 2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: Updating configuration internals Parsing command line Loading file /t2k.org/user/k/sophie.king/what/what.cfg Could not load file /t2k.org/user/k/sophie.king/what/what.cfg: Can't load a cfg file '/t2k.org/user/k/sophie.king/what/what.cfg' Loading file ./what.cfg CFG merged Updating configuration internals Trying to load Resources.LogBackends.StdoutBackend Trying to load DIRAC.Resources.LogBackends.StdoutBackend 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Refreshing configuration... 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Refreshing from list ['dips:// dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server', 'dips:// dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server', 'dips:// diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server'] 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Randomized server list is dips:// dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server, dips:// diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server, dips:// dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Already given a valid url dips:// dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to refresh from dips:// dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to connect to: dips:// dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Using proxy /tmp/x509up_u32226 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: CAs location is /etc/grid-security/certificates 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Loaded 92 CAs [184 CRLs] 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Connection timeout set to: 1 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Authenticated peer (/C=UK/O=eScience/OU=Imperial/L=Physics/CN= dirac02.grid.hep.ph.ic.ac.uk) 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New session connecting to server at ('dirac02.grid.hep.ph.ic.ac.uk', 9135) 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Connected to: dips:// dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New connection -> 146.179.232.11:9135 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Closing socket 2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New version available Updating to version 2020-04-12 18:53:18.780475... 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Updating configuration internals 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Updated to version 2020-04-12 18:53:18.780475 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load Resources.LogBackends.StdoutBackend 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load WebAppDIRAC.Resources.LogBackends.StdoutBackend 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load GridPPDIRAC.Resources.LogBackends.StdoutBackend 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load VMDIRAC.Resources.LogBackends.StdoutBackend 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load DIRAC.Resources.LogBackends.StdoutBackend 2020-04-12 19:27:17 UTC dirac-dms-add-file/FileCatalogFactory [140440172357376] DEBUG: Creating FileCatalog client of type FileCatalog 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load Resources.Catalog.FileCatalogClient 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load WebAppDIRAC.Resources.Catalog.FileCatalogClient 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load GridPPDIRAC.Resources.Catalog.FileCatalogClient 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load VMDIRAC.Resources.Catalog.FileCatalogClient 2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load DIRAC.Resources.Catalog.FileCatalogClient 2020-04-12 19:27:17 UTC dirac-dms-add-file/FileCatalogFactory [140440172357376] DEBUG: Loaded module FileCatalogClient
Keep local file as .cfg but change lfn to .txt - get an error
king@heppc105:~ $ dirac-dms-add-file / t2k.org/user/k/sophie.king/what/what.txt ./what.cfg CA-SFU-T21-disk Traceback (most recent call last): File "/.data/king/grid/dirac/dirac_ui/DIRAC/DataManagementSystem/scripts/dirac-dms-add-file.py", line 71, in <module> lfns.append( getDict( args ) ) File "/.data/king/grid/dirac/dirac_ui/DIRAC/DataManagementSystem/scripts/dirac-dms-add-file.py", line 52, in getDict lfn_dict['SE'] = item_list[2] IndexError: list index out of range
Ensure both the local file and the lfn have .txt - Finally, this does work
king@heppc105:~ $ mv what.cfg what.txt king@heppc105:~ $ dirac-dms-add-file / t2k.org/user/k/sophie.king/what/what.txt ./what.txt CA-SFU-T21-disk
Uploading /t2k.org/user/k/sophie.king/what/what.txt Successfully uploaded file to CA-SFU-T21-disk
I can reproduce this problem with multiple tests.
We can do a workaround of course, but I was wondering if this a known thing, that DIRAC doesnt like the local or LFN file to have .cfg ? And can it be fixed or is it the by-product of something deeper and we must just live with it?
It would seem quite sensible/common that one may have some files on the grid, and for bookkeeping purposes want to preserve a file that ends in .cfg that was used as input to the software that created said files.
Cheers Sophie
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from my guinea pig living room
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: Working from home, please use email. http://www.hep.ph.ic.ac.uk/~dbauer/
-- Sent from my guinea pig living room ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: Working from home, please use email. http://www.hep.ph.ic.ac.uk/~dbauer/