--Hello Daniela,
Thank you for your email. That is correct, Rob has helped me with the first steps. Yes, I have reviewed the earlier thread and it was likely me Rob was talking about. I am trying to run a big cosmic shear simulation for Euclid, something like ~3,000,000 jobs and I hope gridPP will help me achieve that challenging goal! The vast majority of the jobs complete in a few hours, but some (hard to predict) will sometimes take a few days, but a 7 day walltime should be okay.
Thanks for suggesting the guide. Initially I thought I was in a fresh terminal, but hit an issue with the paths. Afterwards I managed to submit and run a job:
$ dirac-wms-job-status -f logfileSo good progress!
JobID=34498902 Status=Done; MinorStatus=Execution Complete; Site=LCG.RAL-LCG2.uk;I noticed that dirac_ui uses Python 2, so had to quick and dirty symlink my system Python 3 local/bin/python directory (I am on Ubuntu 20.04). Unfortunately, my submission script and all my code is in Python 3. Any ideas there?
Regarding the submission, do you recommend trying again the diracos Python API? I think the issues might have been the missing "-S GridPP -C dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server" and "-g [your_vo_goes_here]_user". Using the diracos API might be easier as it supports Python 3 and I can install all my dependencies/code very easily.
Thanks again for you help
Giuseppe
On 28/06/2022 20:38, Daniela Bauer wrote:
This email was sent to you by someone outside the University.You should only click on links or attachments if you are certain that the email is genuine and the content is safe.Hi Giuseppe,
I gather you are the gridpp user Rob Currie alluded to in an earlier email ?If so, welcome aboard.DIRAC installations differ slightly from each other, you can find any setting specific to the GridPP DIRAC instance here:Would you mind trying this out ?
If you have an SL7 machine with cvmfs mounted to hand, I would recommend using the cvmfs based DIRAC UI (in a clean window!!), much quicker to test.We have no access to the ganga setup, which makes debugging a bit difficult, so we generally don't recommend this to beginners.
I checked on our DIRAC server and it looks like you are properly registered, so there shouldn't be a problem there.
As for setting up any software inside your job:This depends on the size of the executable (not everything can be shipped with a sandbox) and other parameters.We have no control over were sites run their jobs, so if you write any setup it has to be relative to $PWD, there is no such thing as a home directory.
Once you managed to submit a job or three - you might want to send a script with it that dumps the environment, so you get a feel for the look of a grid environment - we'll get onto the next step, as in how to get your software where it's meant to go.
Hope that helps,Daniela
On Tue, 28 Jun 2022 at 19:37, Giuseppe Congedo <giuseppe.congedo@ed.ac.uk> wrote:
*******************
This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender.
If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address.
*******************
Hello everyone,
I am relatively new to gridPP, having obtained a certificate/VO
membership only recently. I am writing because I have started
experimenting with the Python DIRAC API (followed the official
instructions
https://dirac.readthedocs.io/en/latest/UserGuide/GettingStarted/InstallingClient/index.html
)
Apart one lucky job (which I managed to submit via
/cvmfs/ganga.cern.ch/dirac_ui/bashrc as opposed to the manual procedure
above, but failed probably due to me deleting the proxy), unfortunately
all my other attempts have failed so far. This is the error I always get:
> Job submission failure Cannot get URL for
> WorkloadManagement/JobManager in setup DIRAC-Certification:
> RuntimeError('Option
> /DIRAC/Setups/DIRAC-Certification/WorkloadManagement is not defined')
I wonder if I am doing something wrong somewere.
Unrelated from the above, I was also wondering what environment the code
will see when it arrives at the remote node. Can I sandbox my Python
package, cd into the directory and install it along with all
dependencies all in a job step? For instance:
> job.setExecutable('cd my_package_dir && pip install -r
> requirements.txt --user && python setup.py install --user')
> job.setExecutable('my_package_dir/my_package/my_script.py')
> job.setInputSandbox([my_package_dir, other_files])
Also, are directories relative to the home?
Many thanks for all your help
Giuseppe
--
Dr Giuseppe Congedo
(Senior Researcher)
Institute for Astronomy, University of Edinburgh
Royal Observatory, Blackford Hill
Edinburgh, EH9 3HJ
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
--
-----------------------------------------------------------
daniela.bauer@imperial.ac.uk
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: Working from home, please use email.
http://www.hep.ph.ic.ac.uk/~dbauer/
-- Dr Giuseppe Congedo (Senior Researcher) Institute for Astronomy, University of Edinburgh Royal Observatory, Blackford Hill Edinburgh, EH9 3HJ
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users