Hi Sophie,

we are trying to get this fixed properly, but it's non-trivial, so it might be some time:
https://github.com/DIRACGrid/DIRAC/pull/4651

Daniela


On Wed, 24 Jun 2020 at 23:41, Sophie King <soph.e.king123@gmail.com> wrote:

Hi Daniela

OK, thanks, that worked!

We don't download the files inside the job, in the case of these particular jobs we write them inside the job and then upload them to the grid for bookkeeping.  Or for other software we may write them with submission scripts and pack in the sandbox.  Or, if a config file is not input-file-dependent, we just put it on cvmfs and all jobs read from there.

Anyway, thanks for the tip, i'll work that into our scripts.

Cheers
Sophie



On 18/06/2020 13:31, Daniela Bauer wrote:
Hi Sophie,

there is a workaround:
You can put all arguments into a file and give that to the command:
lx02:2020_May_27_1434_lz_EL7 > cat uploadme
/gridpp/daniela.bauer/bah/test2.cfg test.cfg UKI-LT2-IC-HEP-disk

lx02:2020_May_27_1434_lz_EL7 > dirac-dms-add-file uploadme

Uploading /gridpp/daniela.bauer/bah/test2.cfg
Successfully uploaded file to UKI-LT2-IC-HEP-disk

and similarly for the download command:
lx02:2020_May_27_1434_lz_EL7 > cat getme
/gridpp/daniela.bauer/bah/test2.cfg

lx02:2020_May_27_1434_lz_EL7 > dirac-dms-get-file getme
{'Failed': {},
 'Successful': {'/gridpp/daniela.bauer/bah/test2.cfg': '/home/hep/dbauer/check_test_dirac/2020_May_27_1434_lz_EL7/test2.cfg'}}

If you are using it inside a dirac job as an input file, it should really go into the inputsandbox as
InputSandbox = {"diractest.sh", "LFN:/gridpp/daniela.bauer/bah/test2.cfg"};
rather than downloading it by hand.

Regards,
Daniela

On Tue, 16 Jun 2020 at 21:31, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Sophie,

%^&*(!!!
You are right. Simon had  a quick look and it  seems to indicate that there is no easy solution to this. 
We'll get back to you (possible after a detour via the DIRAC developers list.)

Regards,
Daniela

On Tue, 16 Jun 2020 at 19:32, Sophie King <soph.e.king123@gmail.com> wrote:

This email from soph.e.king123@gmail.com originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list to disable email stamping for this address.

 

Hi Everyone

As far as I can tell, DIRAC does not like me uploading files that end in .cfg to the DFC.
Below you can see my attempts at uploading a file with .cfg and how the problem is fixed if I move it to a .txt file.


Both local and lfn have what.cfg
- nothing happens 
- get no complaint, but if I check, the file has not uploaded

king@heppc105:~ $ dirac-dms-add-file /t2k.org/user/k/sophie.king/what/what.cfg ./what.cfg CA-SFU-T21-disk


Try again with debug
- no error, just seems to stop, and still no upload

king@heppc105:~ $ dirac-dms-add-file /t2k.org/user/k/sophie.king/what/what.cfg ./what.cfg CA-SFU-T21-disk -ddd

2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: dirac.cfg should be at /.data/king/grid/dirac/dirac_ui/etc/dirac.cfg
2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: CFG merged
2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: Updating configuration internals
2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: Updating configuration internals
Parsing command line
Loading file /t2k.org/user/k/sophie.king/what/what.cfg
Could not load file /t2k.org/user/k/sophie.king/what/what.cfg: Can't load a cfg file '/t2k.org/user/k/sophie.king/what/what.cfg'
Loading file ./what.cfg
CFG merged
Updating configuration internals
Trying to load Resources.LogBackends.StdoutBackend
Trying to load DIRAC.Resources.LogBackends.StdoutBackend
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Refreshing configuration...
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Refreshing from list ['dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server', 'dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server', 'dips://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server']
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Randomized server list is dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server, dips://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server, dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Already given a valid url dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:  Trying to refresh from dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to connect to: dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Using proxy /tmp/x509up_u32226
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: CAs location is /etc/grid-security/certificates
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Loaded 92 CAs [184 CRLs]
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Connection timeout set to:  1
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG:  Authenticated peer (/C=UK/O=eScience/OU=Imperial/L=Physics/CN=dirac02.grid.hep.ph.ic.ac.uk)
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New session connecting to server at ('dirac02.grid.hep.ph.ic.ac.uk', 9135)
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Connected to: dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New connection -> 146.179.232.11:9135
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: Closing socket
2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376] DEBUG: New version available Updating to version 2020-04-12 18:53:18.780475...
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Updating configuration internals
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Updated to version 2020-04-12 18:53:18.780475
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load Resources.LogBackends.StdoutBackend
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load WebAppDIRAC.Resources.LogBackends.StdoutBackend
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load GridPPDIRAC.Resources.LogBackends.StdoutBackend
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load VMDIRAC.Resources.LogBackends.StdoutBackend
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load DIRAC.Resources.LogBackends.StdoutBackend
2020-04-12 19:27:17 UTC dirac-dms-add-file/FileCatalogFactory [140440172357376] DEBUG: Creating FileCatalog client of type FileCatalog
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load Resources.Catalog.FileCatalogClient
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load WebAppDIRAC.Resources.Catalog.FileCatalogClient
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load GridPPDIRAC.Resources.Catalog.FileCatalogClient
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load VMDIRAC.Resources.Catalog.FileCatalogClient
2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376] DEBUG: Trying to load DIRAC.Resources.Catalog.FileCatalogClient
2020-04-12 19:27:17 UTC dirac-dms-add-file/FileCatalogFactory [140440172357376] DEBUG: Loaded module FileCatalogClient


Keep local file as .cfg but change lfn to .txt
- get an error

king@heppc105:~ $ dirac-dms-add-file /t2k.org/user/k/sophie.king/what/what.txt ./what.cfg CA-SFU-T21-disk
Traceback (most recent call last):
  File "/.data/king/grid/dirac/dirac_ui/DIRAC/DataManagementSystem/scripts/dirac-dms-add-file.py", line 71, in <module>
    lfns.append( getDict( args ) )
  File "/.data/king/grid/dirac/dirac_ui/DIRAC/DataManagementSystem/scripts/dirac-dms-add-file.py", line 52, in getDict
    lfn_dict['SE'] = item_list[2]
IndexError: list index out of range


Ensure both the local file and the lfn have .txt
- Finally, this does work

king@heppc105:~ $ mv what.cfg what.txt
king@heppc105:~ $ dirac-dms-add-file /t2k.org/user/k/sophie.king/what/what.txt ./what.txt CA-SFU-T21-disk

Uploading /t2k.org/user/k/sophie.king/what/what.txt
Successfully uploaded file to CA-SFU-T21-disk


I can reproduce this problem with multiple tests.

We can do a workaround of course, but I was wondering if this a known thing, that DIRAC doesnt like the local or LFN file to have .cfg ?   And can it be fixed or is it the by-product of something deeper and we must just live with it?

It would seem quite sensible/common that one may have some files on the grid, and for bookkeeping purposes want to preserve a file that ends in .cfg that was used as input to the software that created said files.

Cheers
Sophie





--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users


--
Sent from my guinea pig living room

-----------------------------------------------------------
daniela.bauer@imperial.ac.uk
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: Working from home, please use email.
http://www.hep.ph.ic.ac.uk/~dbauer/


--
Sent from my guinea pig living room

-----------------------------------------------------------
daniela.bauer@imperial.ac.uk
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: Working from home, please use email.
http://www.hep.ph.ic.ac.uk/~dbauer/

--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users


--
Sent from my guinea pig living room

-----------------------------------------------------------
daniela.bauer@imperial.ac.uk
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: Working from home, please use email.
http://www.hep.ph.ic.ac.uk/~dbauer/