Re: [Gridpp-Dirac-Users] DFC and files ending in .cfg

24 Jun 2020

      Hi Daniela

OK, thanks, that worked!

We don't download the files inside the job, in the case of these 
particular jobs we write them inside the job and then upload them to the 
grid for bookkeeping.  Or for other software we may write them with 
submission scripts and pack in the sandbox.  Or, if a config file is not 
input-file-dependent, we just put it on cvmfs and all jobs read from there.

Anyway, thanks for the tip, i'll work that into our scripts.

Cheers
Sophie

On 18/06/2020 13:31, Daniela Bauer wrote:
...
Hi Sophie,
there is a workaround:
You can put all arguments into a file and give that to the command:
lx02:2020_May_27_1434_lz_EL7 > cat uploadme
/gridpp/daniela.bauer/bah/test2.cfg test.cfg UKI-LT2-IC-HEP-disk
lx02:2020_May_27_1434_lz_EL7 > dirac-dms-add-file uploadme
Uploading /gridpp/daniela.bauer/bah/test2.cfg
Successfully uploaded file to UKI-LT2-IC-HEP-disk
and similarly for the download command:
lx02:2020_May_27_1434_lz_EL7 > cat getme
/gridpp/daniela.bauer/bah/test2.cfg
lx02:2020_May_27_1434_lz_EL7 > dirac-dms-get-file getme
{'Failed': {},
 'Successful': {'/gridpp/daniela.bauer/bah/test2.cfg': 
'/home/hep/dbauer/check_test_dirac/2020_May_27_1434_lz_EL7/test2.cfg'}}
If you are using it inside a dirac job as an input file, it should 
really go into the inputsandbox as
InputSandbox = {"diractest.sh", 
"LFN:/gridpp/daniela.bauer/bah/test2.cfg"};
rather than downloading it by hand.
Regards,
Daniela
On Tue, 16 Jun 2020 at 21:31, Daniela Bauer 
<daniela.bauer.grid@googlemail.com 
<mailto:daniela.bauer.grid@googlemail.com>> wrote:
Hi Sophie,
%^&*(!!!
    You are right. Simon had  a quick look and it  seems to indicate
    that there is no easy solution to this.
    We'll get back to you (possible after a detour via the DIRAC
    developers list.)
Regards,
    Daniela
On Tue, 16 Jun 2020 at 19:32, Sophie King
    <soph.e.king123@gmail.com <mailto:soph.e.king123@gmail.com>> wrote:
This email from soph.e.king123@gmail.com
        <mailto:soph.e.king123@gmail.com> originates from outside
        Imperial. Do not click on links and attachments unless you
        recognise the sender. If you trust the sender, add them to
        your safe senders list
        <https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable
        email stamping for this address.
Hi Everyone
As far as I can tell, DIRAC does not like me uploading files
        that end in .cfg to the DFC.
        Below you can see my attempts at uploading a file with .cfg
        and how the problem is fixed if I move it to a .txt file.
Both local and lfn have what.cfg
        - nothing happens
        - get no complaint, but if I check, the file has not uploaded
king@heppc105:~ $ dirac-dms-add-file
        /t2k.org/user/k/sophie.king/what/what
        <http://t2k.org/user/k/sophie.king/what/what>*.cfg*
        ./what*.cfg* CA-SFU-T21-disk
Try again with debug
        - no error, just seems to stop, and still no upload
king@heppc105:~ $ dirac-dms-add-file
        /t2k.org/user/k/sophie.king/what/what
        <http://t2k.org/user/k/sophie.king/what/what>*.cfg*
        ./what*.cfg* CA-SFU-T21-disk -ddd
2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG:
        dirac.cfg should be at
        /.data/king/grid/dirac/dirac_ui/etc/dirac.cfg
        2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG: CFG
        merged
        2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG:
        Updating configuration internals
        2020-04-12 19:27:16 UTC Framework [140440172357376] DEBUG:
        Updating configuration internals
        Parsing command line
        Loading file /t2k.org/user/k/sophie.king/what/what.cfg
        <http://t2k.org/user/k/sophie.king/what/what.cfg>
        Could not load file /t2k.org/user/k/sophie.king/what/what.cfg
        <http://t2k.org/user/k/sophie.king/what/what.cfg>: Can't load
        a cfg file '/t2k.org/user/k/sophie.king/what/what.cfg
        <http://t2k.org/user/k/sophie.king/what/what.cfg>'
        Loading file ./what.cfg
        CFG merged
        Updating configuration internals
        Trying to load Resources.LogBackends.StdoutBackend
        Trying to load DIRAC.Resources.LogBackends.StdoutBackend
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Refreshing configuration...
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Refreshing from list
        ['dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
        <http://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server>',
        'dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
        <http://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server>',
        'dips://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
        <http://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server>']
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Randomized server list is
        dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
        <http://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server>,
        dips://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
        <http://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server>,
        dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
        <http://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server>
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Already given a valid url
        dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
        <http://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server>
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG:  Trying to refresh from
        dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
        <http://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server>
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Trying to connect to:
        dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
        <http://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server>
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Using proxy /tmp/x509up_u32226
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: CAs location is /etc/grid-security/certificates
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Loaded 92 CAs [184 CRLs]
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Connection timeout set to: 1
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG:  Authenticated peer
        (/C=UK/O=eScience/OU=Imperial/L=Physics/CN=dirac02.grid.hep.ph.ic.ac.uk
        <http://dirac02.grid.hep.ph.ic.ac.uk>)
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: New session connecting to server at
        ('dirac02.grid.hep.ph.ic.ac.uk
        <http://dirac02.grid.hep.ph.ic.ac.uk>', 9135)
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Connected to:
        dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
        <http://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server>
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: New connection -> 146.179.232.11:9135
        <http://146.179.232.11:9135>
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Closing socket
        2020-04-12 19:27:16 UTC dirac-dms-add-file [140440172357376]
        DEBUG: New version available Updating to version 2020-04-12
        18:53:18.780475...
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Updating configuration internals
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Updated to version 2020-04-12 18:53:18.780475
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Trying to load Resources.LogBackends.StdoutBackend
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Trying to load
        WebAppDIRAC.Resources.LogBackends.StdoutBackend
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Trying to load
        GridPPDIRAC.Resources.LogBackends.StdoutBackend
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Trying to load VMDIRAC.Resources.LogBackends.StdoutBackend
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Trying to load DIRAC.Resources.LogBackends.StdoutBackend
        2020-04-12 19:27:17 UTC dirac-dms-add-file/FileCatalogFactory
        [140440172357376] DEBUG: Creating FileCatalog client of type
        FileCatalog
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Trying to load Resources.Catalog.FileCatalogClient
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Trying to load
        WebAppDIRAC.Resources.Catalog.FileCatalogClient
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Trying to load
        GridPPDIRAC.Resources.Catalog.FileCatalogClient
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Trying to load VMDIRAC.Resources.Catalog.FileCatalogClient
        2020-04-12 19:27:17 UTC dirac-dms-add-file [140440172357376]
        DEBUG: Trying to load DIRAC.Resources.Catalog.FileCatalogClient
        2020-04-12 19:27:17 UTC dirac-dms-add-file/FileCatalogFactory
        [140440172357376] DEBUG: Loaded module FileCatalogClient
Keep local file as .cfg but change lfn to .txt
        - get an error
king@heppc105:~ $ dirac-dms-add-file
        /t2k.org/user/k/sophie.king/what/what.txt
        <http://t2k.org/user/k/sophie.king/what/what.txt> ./what.cfg
        CA-SFU-T21-disk
        Traceback (most recent call last):
          File
        "/.data/king/grid/dirac/dirac_ui/DIRAC/DataManagementSystem/scripts/dirac-dms-add-file.py",
        line 71, in <module>
            lfns.append( getDict( args ) )
          File
        "/.data/king/grid/dirac/dirac_ui/DIRAC/DataManagementSystem/scripts/dirac-dms-add-file.py",
        line 52, in getDict
            lfn_dict['SE'] = item_list[2]
        IndexError: list index out of range
Ensure both the local file and the lfn have .txt
        - Finally, this does work
king@heppc105:~ $ mv what.cfg what.txt
        king@heppc105:~ $ dirac-dms-add-file
        /t2k.org/user/k/sophie.king/what/what.txt
        <http://t2k.org/user/k/sophie.king/what/what.txt> ./what.txt
        CA-SFU-T21-disk
Uploading /t2k.org/user/k/sophie.king/what/what.txt
        <http://t2k.org/user/k/sophie.king/what/what.txt>
        Successfully uploaded file to CA-SFU-T21-disk
I can reproduce this problem with multiple tests.
We can do a workaround of course, but I was wondering if this
        a known thing, that DIRAC doesnt like the local or LFN file to
        have .cfg ?   And can it be fixed or is it the by-product of
        something deeper and we must just live with it?
It would seem quite sensible/common that one may have some
        files on the grid, and for bookkeeping purposes want to
        preserve a file that ends in .cfg that was used as input to
        the software that created said files.
Cheers
        Sophie
-- 
        _______________________________________________
        Gridpp-Dirac-Users mailing list
        Gridpp-Dirac-Users@imperial.ac.uk
        <mailto:Gridpp-Dirac-Users@imperial.ac.uk>
        https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- 
    Sent from my guinea pig living room
-----------------------------------------------------------
    daniela.bauer@imperial.ac.uk <mailto:daniela.bauer@imperial.ac.uk>
    HEP Group/Physics Dep
    Imperial College
    London, SW7 2BW
    Tel: Working from home, please use email.
    http://www.hep.ph.ic.ac.uk/~dbauer/
-- 
Sent from my guinea pig living room
-----------------------------------------------------------
daniela.bauer@imperial.ac.uk <mailto:daniela.bauer@imperial.ac.uk>
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: Working from home, please use email.
http://www.hep.ph.ic.ac.uk/~dbauer/