Transferring Directories to DIRAC
******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Good afternoon, Could you advise me on the best way to transfer a directory to DIRAC? I want to use the neutron cross section library JEFF-3.3 for a simulation I'm running but it's too big to include in the sandbox (~1.9G) when I submit the jobs and is a directory with quite a lot of files so I can't just use dirac-dms-add-file. I've had a quick look on cvmfs and I can't find it available on there as an alternative. Thanks for your help, Kind regards, Jo Orpwood
Hi Jo, On Fri, Apr 14, 2023 at 01:26:32PM +0100, Jonathan Orpwood wrote:
Could you advise me on the best way to transfer a directory to DIRAC?
It's possible to use dirac-dms-directory-sync to upload (or download) an entire directory: dirac-dms-directory-sync local_path LFN_path SE The problem with doing that is that it's then difficult to reference a large selection of files in a job, although it may be viable if you only need a couple of the files in each job. If you need the whole (or most of the) reference set in every job, it would be better to create a single large .tar file containing them all and upload that with dirac-dms-add-file. You can then reference it in the input sandbox by LFN to get it copied to the WN (DIRAC will probably unpack this for your automatically in the job's default scratch location). If you're planning on running a large number of jobs, we may want to see if we can get the reference dataset copied onto a suitable CVMFS area instead. Regards, Simon
participants (2)
-
Jonathan Orpwood
-
Simon Fayer