Dirac job submission problem [new user]
Hello all, I am trying to move my workflow on gridpp, but the problem is, I am a newbie! Andrew Lahiff has been very helpful in giving me good advice along the way - thanks. I hope you too could help me with this. I set up code, software environment, and data, and it works fine locally on Ganga. To move things on the grid, I put code & environment on cvmfs (thanks Catalin Condurache), and uploaded data on DFC. After submission, I got the error attached below. It looks like the job could not see the input data, and also some output sandbox was not properly initialised. My submission scripts looks like the example in the userguide. I suspect I might be doing something wrong perhaps here? j.inputfiles = [ DiracFile('LFN:/gridpp/user/giuseppe.congedo/my_folder/my_file'), Please note that the job fails no matter if I set DiracLFNBase or not in .gangarc. I look forward to hearing from you, and thanks so much for your help. Giuseppe --- ERROR Unable to finalise job after 36 retries due to error: GangaDiracError: No Output sandbox registered for job 2335379 INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (2 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (3 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (4 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (5 of 5) after 2.5-sec delay ERROR GangaDiracError: No Output sandbox registered for job 2335379 --- -- Dr Giuseppe Congedo (Postdoctoral Research Associate) Institute of Astronomy, Royal Observatory University of Edinburgh Blackford Hill, Edinburgh EH9 3HJ +44 131 6688 350 The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Hi Giuseppe, You can generally get more details information about why a job failed from the web interface at: https://dirac.gridpp.ac.uk (Although you have to have your grid certificate installed in your browser to access this). In this case your jobs have the Application Status of "Input data not available", which as you correctly surmised indicates a problem with the inputfiles. I've had a look at the JDL which Ganga generated and you have three LFN based files in the sandbox: "LFN:/gridpp/user/giuseppe.congedo/sim/image_0_details.fits", "LFN:/gridpp/user/giuseppe.congedo/PSF/simssbc_cb2004a_001.fits_0.000_0.804_1.00.fits", "LFN:/gridpp/user/giuseppe.congedo/sim/.lensmc_cache", If I look in the catalogue (using the dirac-dms-filecatalog-cli tool), it seems like you used a slightly different path for these files when you uploaded them: /gridpp/user/giuseppe.congedo/sim/sim/image_0_details.fits /gridpp/user/giuseppe.congedo/sim/PSF/simssbc_cb2004a_001.fits_0.000_0.804_1.00.fits /gridpp/user/giuseppe.congedo/sim/.lensmc_cache (Note the extra "sim/" directory in the first two paths). It should simply be a case of adjusting the paths in your ganga script to match the file catalogue. It also looks like you may have put these as inputdata as well... There is no need to do this, just specifying them in the inputfiles is sufficient (if you set inputdata, the jobs have to run at a site with those files, whereas inputfiles can be downloaded at any site). Please let us know if that doesn't work or if you run into any further problems. Regards, Simon On Sat, Mar 18, 2017 at 11:21:56AM +0000, Giuseppe Congedo wrote:
Hello all,
I am trying to move my workflow on gridpp, but the problem is, I am a newbie! Andrew Lahiff has been very helpful in giving me good advice along the way - thanks. I hope you too could help me with this.
I set up code, software environment, and data, and it works fine locally on Ganga. To move things on the grid, I put code & environment on cvmfs (thanks Catalin Condurache), and uploaded data on DFC. After submission, I got the error attached below. It looks like the job could not see the input data, and also some output sandbox was not properly initialised.
My submission scripts looks like the example in the userguide. I suspect I might be doing something wrong perhaps here?
j.inputfiles = [ DiracFile('LFN:/gridpp/user/giuseppe.congedo/my_folder/my_file'),
Please note that the job fails no matter if I set DiracLFNBase or not in .gangarc.
I look forward to hearing from you, and thanks so much for your help.
Giuseppe
--- ERROR Unable to finalise job after 36 retries due to error: GangaDiracError: No Output sandbox registered for job 2335379 INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (2 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (3 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (4 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (5 of 5) after 2.5-sec delay ERROR GangaDiracError: No Output sandbox registered for job 2335379 ---
Hi Simon, Thanks so much for looking into this. I've fixed up the paths - thanks for spotting this. It now looks like the job got somewhere nearer to start, but again it's been put on hold because it doesn't see the files - please find the log below. Also, I've been looking at the JDL and I do have indeed both an InputSandbox section (with LFN paths to files), and an InputData section (with the same files) too. I suspect it's exactly the problem you mentioned. In fact, two identical jobs were both submitted to LCG.UKI-SCOTGRID-ECDF.uk where my data live. Please look at job no. 2335476 and 2335466. I'm puzzled by the fact that I only set this: j.inputfiles = [ DiracFile(... in my ganga job sumission file. Is this correct? If so, why is Dirac trying to use InputData? Thanks again for all your useful tips. Giuseppe --- JobManager Received Job accepted Unknown 2017-03-18 13:12 JobPath Checking JobSanity Unknown 2017-03-18 13:12 JobSanity Checking InputData Unknown 2017-03-18 13:12 InputData Checking JobScheduling Unknown 2017-03-18 13:12 JobScheduling Waiting Pilot Agent Submission Unknown 2017-03-18 13:12 Matcher Matched Assigned Unknown 2017-03-18 13:16 JobAgent@LCG.UKI-SCOTGRID-ECDF.u Matched Job Received by Agent Unknown 2017-03-18 13:16 JobAgent@LCG.UKI-SCOTGRID-ECDF.u Matched Submitted To CE Unknown 2017-03-18 13:16 JobWrapper Running Job Initialization Unknown 2017-03-18 13:16 JobWrapper Running Downloading InputSandbox Unknown 2017-03-18 13:16 JobWrapper Running Downloading InputSandbox LFN(s) Unknown 2017-03-18 13:16 JobWrapper Running Application Failed Input Sandbox Download 2017-03-18 13:16 JobWrapper Rescheduled Input Sandbox Download Failed Input Sandbox Download 2017-03-18 13:16 JobManager Received Job Rescheduled Unknown 2017-03-18 13:16 JobPath Checking JobSanity Unknown 2017-03-18 13:16 JobSanity Checking InputData Unknown 2017-03-18 13:16 InputData Checking JobScheduling Unknown 2017-03-18 13:16 JobScheduling Checking JobScheduling On Hold: after rescheduling 1 2017-03-18 13:16 JobScheduling Waiting Pilot Agent Submission Unknown 2017-03-18 13:20 Matcher Matched Assigned Unknown 2017-03-18 13:24 JobAgent@LCG.UKI-SCOTGRID-ECDF.u Matched Job Received by Agent Unknown 2017-03-18 13:24 JobAgent@LCG.UKI-SCOTGRID-ECDF.u Matched Submitted To CE Unknown 2017-03-18 13:24 JobWrapper Running Job Initialization Unknown 2017-03-18 13:24 JobWrapper Running Downloading InputSandbox Unknown 2017-03-18 13:24 JobWrapper Running Downloading InputSandbox LFN(s) Unknown 2017-03-18 13:24 JobWrapper Running Application Failed Input Sandbox Download 2017-03-18 13:24 JobWrapper Rescheduled Input Sandbox Download Failed Input Sandbox Download 2017-03-18 13:24 JobManager Received Job Rescheduled Unknown 2017-03-18 13:24 JobPath Checking JobSanity Unknown 2017-03-18 13:24 JobSanity Checking InputData Unknown 2017-03-18 13:24 InputData Checking JobScheduling Unknown 2017-03-18 13:24 JobScheduling Checking JobScheduling On Hold: after rescheduling 2 2017-03-18 13:24 JobScheduling Waiting Pilot Agent Submission Unknown 2017-03-18 13:30 Matcher Matched Assigned Unknown 2017-03-18 13:34 JobAgent@LCG.UKI-SCOTGRID-ECDF.u Matched Job Received by Agent Unknown 2017-03-18 13:34 JobAgent@LCG.UKI-SCOTGRID-ECDF.u Matched Submitted To CE Unknown 2017-03-18 13:34 JobWrapper Running Job Initialization Unknown 2017-03-18 13:34 JobWrapper Running Downloading InputSandbox Unknown 2017-03-18 13:34 JobWrapper Running Downloading InputSandbox LFN(s) Unknown 2017-03-18 13:34 JobWrapper Running Application Failed Input Sandbox Download 2017-03-18 13:34 JobWrapper Rescheduled Input Sandbox Download Failed Input Sandbox Download 2017-03-18 13:34 JobManager Received Job Rescheduled Unknown 2017-03-18 13:34 JobPath Checking JobSanity Unknown 2017-03-18 13:34 JobSanity Checking InputData Unknown 2017-03-18 13:34 InputData Checking JobScheduling Unknown 2017-03-18 13:34 JobScheduling Checking JobScheduling On Hold: after rescheduling 3 2017-03-18 13:34 JobScheduling Waiting Pilot Agent Submission Unknown 2017-03-18 13:45 Matcher Matched Assigned Unknown 2017-03-18 13:48 JobAgent@LCG.UKI-SCOTGRID-ECDF.u Matched Job Received by Agent Unknown 2017-03-18 13:48 JobAgent@LCG.UKI-SCOTGRID-ECDF.u Matched Submitted To CE Unknown 2017-03-18 13:48 JobWrapper Running Job Initialization Unknown 2017-03-18 13:48 JobWrapper Running Downloading InputSandbox Unknown 2017-03-18 13:48 JobWrapper Running Downloading InputSandbox LFN(s) Unknown 2017-03-18 13:48 JobWrapper Running Application Failed Input Sandbox Download 2017-03-18 13:48 JobWrapper Rescheduled Input Sandbox Download Failed Input Sandbox Download 2017-03-18 13:48 JobManager Received Job Rescheduled Unknown 2017-03-18 13:48 JobPath Checking JobSanity Unknown 2017-03-18 13:48 JobSanity Checking InputData Unknown 2017-03-18 13:48 InputData Checking JobScheduling Unknown 2017-03-18 13:48 JobScheduling Checking JobScheduling On Hold: after rescheduling 4 2017-03-18 13:48 JobScheduling Waiting Pilot Agent Submission Unknown 2017-03-18 13:59 Matcher Matched Assigned Unknown 2017-03-18 14:02 JobAgent@LCG.UKI-SCOTGRID-ECDF.u Matched Job Received by Agent Unknown 2017-03-18 14:02 JobAgent@LCG.UKI-SCOTGRID-ECDF.u Matched Submitted To CE Unknown 2017-03-18 14:02 JobWrapper Running Job Initialization Unknown 2017-03-18 14:02 JobWrapper Running Downloading InputSandbox Unknown 2017-03-18 14:02 JobWrapper Running Downloading InputSandbox LFN(s) Unknown 2017-03-18 14:02 JobWrapper Running Application Failed Input Sandbox Download 2017-03-18 14:02 JobWrapper Rescheduled Input Sandbox Download Failed Input Sandbox Download 2017-03-18 14:02 JobManager Received Job Rescheduled Unknown 2017-03-18 14:02 JobPath Checking JobSanity Unknown 2017-03-18 14:02 JobSanity Checking InputData Unknown 2017-03-18 14:02 InputData Checking JobScheduling Unknown 2017-03-18 14:02 JobScheduling Checking JobScheduling On Hold: after rescheduling 5 2017-03-18 14:02 --- On 18/03/2017 12:33, Simon Fayer wrote:
Hi Giuseppe,
You can generally get more details information about why a job failed from the web interface at: https://dirac.gridpp.ac.uk (Although you have to have your grid certificate installed in your browser to access this).
In this case your jobs have the Application Status of "Input data not available", which as you correctly surmised indicates a problem with the inputfiles.
I've had a look at the JDL which Ganga generated and you have three LFN based files in the sandbox:
"LFN:/gridpp/user/giuseppe.congedo/sim/image_0_details.fits", "LFN:/gridpp/user/giuseppe.congedo/PSF/simssbc_cb2004a_001.fits_0.000_0.804_1.00.fits", "LFN:/gridpp/user/giuseppe.congedo/sim/.lensmc_cache",
If I look in the catalogue (using the dirac-dms-filecatalog-cli tool), it seems like you used a slightly different path for these files when you uploaded them:
/gridpp/user/giuseppe.congedo/sim/sim/image_0_details.fits /gridpp/user/giuseppe.congedo/sim/PSF/simssbc_cb2004a_001.fits_0.000_0.804_1.00.fits /gridpp/user/giuseppe.congedo/sim/.lensmc_cache
(Note the extra "sim/" directory in the first two paths). It should simply be a case of adjusting the paths in your ganga script to match the file catalogue.
It also looks like you may have put these as inputdata as well... There is no need to do this, just specifying them in the inputfiles is sufficient (if you set inputdata, the jobs have to run at a site with those files, whereas inputfiles can be downloaded at any site).
Please let us know if that doesn't work or if you run into any further problems.
Regards, Simon
On Sat, Mar 18, 2017 at 11:21:56AM +0000, Giuseppe Congedo wrote:
Hello all,
I am trying to move my workflow on gridpp, but the problem is, I am a newbie! Andrew Lahiff has been very helpful in giving me good advice along the way - thanks. I hope you too could help me with this.
I set up code, software environment, and data, and it works fine locally on Ganga. To move things on the grid, I put code & environment on cvmfs (thanks Catalin Condurache), and uploaded data on DFC. After submission, I got the error attached below. It looks like the job could not see the input data, and also some output sandbox was not properly initialised.
My submission scripts looks like the example in the userguide. I suspect I might be doing something wrong perhaps here?
j.inputfiles = [ DiracFile('LFN:/gridpp/user/giuseppe.congedo/my_folder/my_file'),
Please note that the job fails no matter if I set DiracLFNBase or not in .gangarc.
I look forward to hearing from you, and thanks so much for your help.
Giuseppe
--- ERROR Unable to finalise job after 36 retries due to error: GangaDiracError: No Output sandbox registered for job 2335379 INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (2 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (3 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (4 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (5 of 5) after 2.5-sec delay ERROR GangaDiracError: No Output sandbox registered for job 2335379 ---
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Hi, I've had the same problem with Ganga, adding the following to your job submission seems to make it only add InputSandbox to the JDL: j.backend.settings['InputData'] = '' Will. On 18/03/2017 14:28, Giuseppe Congedo wrote:
Hi Simon,
Thanks so much for looking into this. I've fixed up the paths - thanks for spotting this. It now looks like the job got somewhere nearer to start, but again it's been put on hold because it doesn't see the files - please find the log below.
Also, I've been looking at the JDL and I do have indeed both an InputSandbox section (with LFN paths to files), and an InputData section (with the same files) too. I suspect it's exactly the problem you mentioned. In fact, two identical jobs were both submitted to LCG.UKI-SCOTGRID-ECDF.uk where my data live. Please look at job no. 2335476 and 2335466. I'm puzzled by the fact that I only set this:
j.inputfiles = [ DiracFile(...
in my ganga job sumission file. Is this correct? If so, why is Dirac trying to use InputData?
Thanks again for all your useful tips.
Giuseppe
--- JobManager
Received
Job accepted
Unknown
2017-03-18 13:12
JobPath
Checking
JobSanity
Unknown
2017-03-18 13:12
JobSanity
Checking
InputData
Unknown
2017-03-18 13:12
InputData
Checking
JobScheduling
Unknown
2017-03-18 13:12
JobScheduling
Waiting
Pilot Agent Submission
Unknown
2017-03-18 13:12
Matcher
Matched
Assigned
Unknown
2017-03-18 13:16
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Job Received by Agent
Unknown
2017-03-18 13:16
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Submitted To CE
Unknown
2017-03-18 13:16
JobWrapper
Running
Job Initialization
Unknown
2017-03-18 13:16
JobWrapper
Running
Downloading InputSandbox
Unknown
2017-03-18 13:16
JobWrapper
Running
Downloading InputSandbox LFN(s)
Unknown
2017-03-18 13:16
JobWrapper
Running
Application
Failed Input Sandbox Download
2017-03-18 13:16
JobWrapper
Rescheduled
Input Sandbox Download
Failed Input Sandbox Download
2017-03-18 13:16
JobManager
Received
Job Rescheduled
Unknown
2017-03-18 13:16
JobPath
Checking
JobSanity
Unknown
2017-03-18 13:16
JobSanity
Checking
InputData
Unknown
2017-03-18 13:16
InputData
Checking
JobScheduling
Unknown
2017-03-18 13:16
JobScheduling
Checking
JobScheduling
On Hold: after rescheduling 1
2017-03-18 13:16
JobScheduling
Waiting
Pilot Agent Submission
Unknown
2017-03-18 13:20
Matcher
Matched
Assigned
Unknown
2017-03-18 13:24
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Job Received by Agent
Unknown
2017-03-18 13:24
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Submitted To CE
Unknown
2017-03-18 13:24
JobWrapper
Running
Job Initialization
Unknown
2017-03-18 13:24
JobWrapper
Running
Downloading InputSandbox
Unknown
2017-03-18 13:24
JobWrapper
Running
Downloading InputSandbox LFN(s)
Unknown
2017-03-18 13:24
JobWrapper
Running
Application
Failed Input Sandbox Download
2017-03-18 13:24
JobWrapper
Rescheduled
Input Sandbox Download
Failed Input Sandbox Download
2017-03-18 13:24
JobManager
Received
Job Rescheduled
Unknown
2017-03-18 13:24
JobPath
Checking
JobSanity
Unknown
2017-03-18 13:24
JobSanity
Checking
InputData
Unknown
2017-03-18 13:24
InputData
Checking
JobScheduling
Unknown
2017-03-18 13:24
JobScheduling
Checking
JobScheduling
On Hold: after rescheduling 2
2017-03-18 13:24
JobScheduling
Waiting
Pilot Agent Submission
Unknown
2017-03-18 13:30
Matcher
Matched
Assigned
Unknown
2017-03-18 13:34
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Job Received by Agent
Unknown
2017-03-18 13:34
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Submitted To CE
Unknown
2017-03-18 13:34
JobWrapper
Running
Job Initialization
Unknown
2017-03-18 13:34
JobWrapper
Running
Downloading InputSandbox
Unknown
2017-03-18 13:34
JobWrapper
Running
Downloading InputSandbox LFN(s)
Unknown
2017-03-18 13:34
JobWrapper
Running
Application
Failed Input Sandbox Download
2017-03-18 13:34
JobWrapper
Rescheduled
Input Sandbox Download
Failed Input Sandbox Download
2017-03-18 13:34
JobManager
Received
Job Rescheduled
Unknown
2017-03-18 13:34
JobPath
Checking
JobSanity
Unknown
2017-03-18 13:34
JobSanity
Checking
InputData
Unknown
2017-03-18 13:34
InputData
Checking
JobScheduling
Unknown
2017-03-18 13:34
JobScheduling
Checking
JobScheduling
On Hold: after rescheduling 3
2017-03-18 13:34
JobScheduling
Waiting
Pilot Agent Submission
Unknown
2017-03-18 13:45
Matcher
Matched
Assigned
Unknown
2017-03-18 13:48
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Job Received by Agent
Unknown
2017-03-18 13:48
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Submitted To CE
Unknown
2017-03-18 13:48
JobWrapper
Running
Job Initialization
Unknown
2017-03-18 13:48
JobWrapper
Running
Downloading InputSandbox
Unknown
2017-03-18 13:48
JobWrapper
Running
Downloading InputSandbox LFN(s)
Unknown
2017-03-18 13:48
JobWrapper
Running
Application
Failed Input Sandbox Download
2017-03-18 13:48
JobWrapper
Rescheduled
Input Sandbox Download
Failed Input Sandbox Download
2017-03-18 13:48
JobManager
Received
Job Rescheduled
Unknown
2017-03-18 13:48
JobPath
Checking
JobSanity
Unknown
2017-03-18 13:48
JobSanity
Checking
InputData
Unknown
2017-03-18 13:48
InputData
Checking
JobScheduling
Unknown
2017-03-18 13:48
JobScheduling
Checking
JobScheduling
On Hold: after rescheduling 4
2017-03-18 13:48
JobScheduling
Waiting
Pilot Agent Submission
Unknown
2017-03-18 13:59
Matcher
Matched
Assigned
Unknown
2017-03-18 14:02
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Job Received by Agent
Unknown
2017-03-18 14:02
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Submitted To CE
Unknown
2017-03-18 14:02
JobWrapper
Running
Job Initialization
Unknown
2017-03-18 14:02
JobWrapper
Running
Downloading InputSandbox
Unknown
2017-03-18 14:02
JobWrapper
Running
Downloading InputSandbox LFN(s)
Unknown
2017-03-18 14:02
JobWrapper
Running
Application
Failed Input Sandbox Download
2017-03-18 14:02
JobWrapper
Rescheduled
Input Sandbox Download
Failed Input Sandbox Download
2017-03-18 14:02
JobManager
Received
Job Rescheduled
Unknown
2017-03-18 14:02
JobPath
Checking
JobSanity
Unknown
2017-03-18 14:02
JobSanity
Checking
InputData
Unknown
2017-03-18 14:02
InputData
Checking
JobScheduling
Unknown
2017-03-18 14:02
JobScheduling
Checking
JobScheduling
On Hold: after rescheduling 5
2017-03-18 14:02
---
On 18/03/2017 12:33, Simon Fayer wrote:
Hi Giuseppe,
You can generally get more details information about why a job failed from the web interface at: https://dirac.gridpp.ac.uk (Although you have to have your grid certificate installed in your browser to access this).
In this case your jobs have the Application Status of "Input data not available", which as you correctly surmised indicates a problem with the inputfiles.
I've had a look at the JDL which Ganga generated and you have three LFN based files in the sandbox:
"LFN:/gridpp/user/giuseppe.congedo/sim/image_0_details.fits", "LFN:/gridpp/user/giuseppe.congedo/PSF/simssbc_cb2004a_001.fits_0.000_0.804_1.00.fits", "LFN:/gridpp/user/giuseppe.congedo/sim/.lensmc_cache",
If I look in the catalogue (using the dirac-dms-filecatalog-cli tool), it seems like you used a slightly different path for these files when you uploaded them:
/gridpp/user/giuseppe.congedo/sim/sim/image_0_details.fits /gridpp/user/giuseppe.congedo/sim/PSF/simssbc_cb2004a_001.fits_0.000_0.804_1.00.fits /gridpp/user/giuseppe.congedo/sim/.lensmc_cache
(Note the extra "sim/" directory in the first two paths). It should simply be a case of adjusting the paths in your ganga script to match the file catalogue.
It also looks like you may have put these as inputdata as well... There is no need to do this, just specifying them in the inputfiles is sufficient (if you set inputdata, the jobs have to run at a site with those files, whereas inputfiles can be downloaded at any site).
Please let us know if that doesn't work or if you run into any further problems.
Regards, Simon
On Sat, Mar 18, 2017 at 11:21:56AM +0000, Giuseppe Congedo wrote:
Hello all,
I am trying to move my workflow on gridpp, but the problem is, I am a newbie! Andrew Lahiff has been very helpful in giving me good advice along the way - thanks. I hope you too could help me with this.
I set up code, software environment, and data, and it works fine locally on Ganga. To move things on the grid, I put code & environment on cvmfs (thanks Catalin Condurache), and uploaded data on DFC. After submission, I got the error attached below. It looks like the job could not see the input data, and also some output sandbox was not properly initialised.
My submission scripts looks like the example in the userguide. I suspect I might be doing something wrong perhaps here?
j.inputfiles = [ DiracFile('LFN:/gridpp/user/giuseppe.congedo/my_folder/my_file'),
Please note that the job fails no matter if I set DiracLFNBase or not in .gangarc.
I look forward to hearing from you, and thanks so much for your help.
Giuseppe
--- ERROR Unable to finalise job after 36 retries due to error: GangaDiracError: No Output sandbox registered for job 2335379 INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (2 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (3 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (4 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (5 of 5) after 2.5-sec delay ERROR GangaDiracError: No Output sandbox registered for job 2335379 ---
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Hi Simon, Thanks so much. That helped indeed, the job was correctly dispatched to a site, and data correctly downloaded, which I think it's already good progress! But it ran into problems afterwards as it crashed a few minutes later. The stdout and stderr don't say that much, so I'm not sure what happened. Also peek in ganga is not very informative, showing only a warning about a non-existent scratch directory. Any clues? Thanks again for your help. Giuseppe ================================================================================== Last 8 lines of application output from JobWrapper on 2017-03-18 14:54:47.985686 : CPU Total: 00:00:02 (h:m:s) Normalized CPU Total 24.7 s @ HEP'06 ================================================================================== 2017-03-18 14:54:39 UTC dirac-jobexec INFO: JobID: 2335477 2017-03-18 14:54:39 UTC dirac-jobexec INFO: DIRAC JobID 2335477 is running at site VAC.UKI-NORTHGRID-MAN-HEP.uk Executing StepInstance RunScriptStep1 of type ScriptStep1 ['ScriptStep1'] 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Command is: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: Non-zero status while executing 2: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Output written to Ganga_Executable.log, execution complete. 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: 'exe-script.py' Exited With Status 2 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: ===== Terminating===== JobManager Received Job accepted Unknown 2017-03-18 14:46 JobPath Checking JobSanity Unknown 2017-03-18 14:46 JobSanity Checking JobScheduling Unknown 2017-03-18 14:46 JobScheduling Waiting Pilot Agent Submission Unknown 2017-03-18 14:46 Matcher Matched Assigned Unknown 2017-03-18 14:54 JobAgent@VAC.UKI-NORTHGRID-MAN-H Matched Job Received by Agent Unknown 2017-03-18 14:54 JobAgent@VAC.UKI-NORTHGRID-MAN-H Matched Submitted To CE Unknown 2017-03-18 14:54 JobWrapper Running Job Initialization Unknown 2017-03-18 14:54 JobWrapper Running Downloading InputSandbox Unknown 2017-03-18 14:54 JobWrapper Running Downloading InputSandbox LFN(s) Unknown 2017-03-18 14:54 JobWrapper Running Application Unknown 2017-03-18 14:54 Job_2335477 Running Application Executing RunScriptStep1 2017-03-18 14:54 Job_2335477 Running Application exe-script.py Exited With Status 2 2017-03-18 14:54 Job_2335477 Running Application exe-script.py Exited With Status 2 2017-03-18 14:54 JobWrapper Completed Application Finished With Errors exe-script.py Exited With Status 2 2017-03-18 14:54 JobWrapper Failed Application Finished With Errors exe-script.py Exited With Status 2 2017-03-18 14:54 On 18/03/2017 14:37, Will Furnell wrote:
Hi,
I've had the same problem with Ganga, adding the following to your job submission seems to make it only add InputSandbox to the JDL:
j.backend.settings['InputData'] = ''
Will.
On 18/03/2017 14:28, Giuseppe Congedo wrote:
Hi Simon,
Thanks so much for looking into this. I've fixed up the paths - thanks for spotting this. It now looks like the job got somewhere nearer to start, but again it's been put on hold because it doesn't see the files - please find the log below.
Also, I've been looking at the JDL and I do have indeed both an InputSandbox section (with LFN paths to files), and an InputData section (with the same files) too. I suspect it's exactly the problem you mentioned. In fact, two identical jobs were both submitted to LCG.UKI-SCOTGRID-ECDF.uk where my data live. Please look at job no. 2335476 and 2335466. I'm puzzled by the fact that I only set this:
j.inputfiles = [ DiracFile(...
in my ganga job sumission file. Is this correct? If so, why is Dirac trying to use InputData?
Thanks again for all your useful tips.
Giuseppe
--- JobManager
Received
Job accepted
Unknown
2017-03-18 13:12
JobPath
Checking
JobSanity
Unknown
2017-03-18 13:12
JobSanity
Checking
InputData
Unknown
2017-03-18 13:12
InputData
Checking
JobScheduling
Unknown
2017-03-18 13:12
JobScheduling
Waiting
Pilot Agent Submission
Unknown
2017-03-18 13:12
Matcher
Matched
Assigned
Unknown
2017-03-18 13:16
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Job Received by Agent
Unknown
2017-03-18 13:16
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Submitted To CE
Unknown
2017-03-18 13:16
JobWrapper
Running
Job Initialization
Unknown
2017-03-18 13:16
JobWrapper
Running
Downloading InputSandbox
Unknown
2017-03-18 13:16
JobWrapper
Running
Downloading InputSandbox LFN(s)
Unknown
2017-03-18 13:16
JobWrapper
Running
Application
Failed Input Sandbox Download
2017-03-18 13:16
JobWrapper
Rescheduled
Input Sandbox Download
Failed Input Sandbox Download
2017-03-18 13:16
JobManager
Received
Job Rescheduled
Unknown
2017-03-18 13:16
JobPath
Checking
JobSanity
Unknown
2017-03-18 13:16
JobSanity
Checking
InputData
Unknown
2017-03-18 13:16
InputData
Checking
JobScheduling
Unknown
2017-03-18 13:16
JobScheduling
Checking
JobScheduling
On Hold: after rescheduling 1
2017-03-18 13:16
JobScheduling
Waiting
Pilot Agent Submission
Unknown
2017-03-18 13:20
Matcher
Matched
Assigned
Unknown
2017-03-18 13:24
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Job Received by Agent
Unknown
2017-03-18 13:24
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Submitted To CE
Unknown
2017-03-18 13:24
JobWrapper
Running
Job Initialization
Unknown
2017-03-18 13:24
JobWrapper
Running
Downloading InputSandbox
Unknown
2017-03-18 13:24
JobWrapper
Running
Downloading InputSandbox LFN(s)
Unknown
2017-03-18 13:24
JobWrapper
Running
Application
Failed Input Sandbox Download
2017-03-18 13:24
JobWrapper
Rescheduled
Input Sandbox Download
Failed Input Sandbox Download
2017-03-18 13:24
JobManager
Received
Job Rescheduled
Unknown
2017-03-18 13:24
JobPath
Checking
JobSanity
Unknown
2017-03-18 13:24
JobSanity
Checking
InputData
Unknown
2017-03-18 13:24
InputData
Checking
JobScheduling
Unknown
2017-03-18 13:24
JobScheduling
Checking
JobScheduling
On Hold: after rescheduling 2
2017-03-18 13:24
JobScheduling
Waiting
Pilot Agent Submission
Unknown
2017-03-18 13:30
Matcher
Matched
Assigned
Unknown
2017-03-18 13:34
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Job Received by Agent
Unknown
2017-03-18 13:34
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Submitted To CE
Unknown
2017-03-18 13:34
JobWrapper
Running
Job Initialization
Unknown
2017-03-18 13:34
JobWrapper
Running
Downloading InputSandbox
Unknown
2017-03-18 13:34
JobWrapper
Running
Downloading InputSandbox LFN(s)
Unknown
2017-03-18 13:34
JobWrapper
Running
Application
Failed Input Sandbox Download
2017-03-18 13:34
JobWrapper
Rescheduled
Input Sandbox Download
Failed Input Sandbox Download
2017-03-18 13:34
JobManager
Received
Job Rescheduled
Unknown
2017-03-18 13:34
JobPath
Checking
JobSanity
Unknown
2017-03-18 13:34
JobSanity
Checking
InputData
Unknown
2017-03-18 13:34
InputData
Checking
JobScheduling
Unknown
2017-03-18 13:34
JobScheduling
Checking
JobScheduling
On Hold: after rescheduling 3
2017-03-18 13:34
JobScheduling
Waiting
Pilot Agent Submission
Unknown
2017-03-18 13:45
Matcher
Matched
Assigned
Unknown
2017-03-18 13:48
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Job Received by Agent
Unknown
2017-03-18 13:48
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Submitted To CE
Unknown
2017-03-18 13:48
JobWrapper
Running
Job Initialization
Unknown
2017-03-18 13:48
JobWrapper
Running
Downloading InputSandbox
Unknown
2017-03-18 13:48
JobWrapper
Running
Downloading InputSandbox LFN(s)
Unknown
2017-03-18 13:48
JobWrapper
Running
Application
Failed Input Sandbox Download
2017-03-18 13:48
JobWrapper
Rescheduled
Input Sandbox Download
Failed Input Sandbox Download
2017-03-18 13:48
JobManager
Received
Job Rescheduled
Unknown
2017-03-18 13:48
JobPath
Checking
JobSanity
Unknown
2017-03-18 13:48
JobSanity
Checking
InputData
Unknown
2017-03-18 13:48
InputData
Checking
JobScheduling
Unknown
2017-03-18 13:48
JobScheduling
Checking
JobScheduling
On Hold: after rescheduling 4
2017-03-18 13:48
JobScheduling
Waiting
Pilot Agent Submission
Unknown
2017-03-18 13:59
Matcher
Matched
Assigned
Unknown
2017-03-18 14:02
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Job Received by Agent
Unknown
2017-03-18 14:02
JobAgent@LCG.UKI-SCOTGRID-ECDF.u
Matched
Submitted To CE
Unknown
2017-03-18 14:02
JobWrapper
Running
Job Initialization
Unknown
2017-03-18 14:02
JobWrapper
Running
Downloading InputSandbox
Unknown
2017-03-18 14:02
JobWrapper
Running
Downloading InputSandbox LFN(s)
Unknown
2017-03-18 14:02
JobWrapper
Running
Application
Failed Input Sandbox Download
2017-03-18 14:02
JobWrapper
Rescheduled
Input Sandbox Download
Failed Input Sandbox Download
2017-03-18 14:02
JobManager
Received
Job Rescheduled
Unknown
2017-03-18 14:02
JobPath
Checking
JobSanity
Unknown
2017-03-18 14:02
JobSanity
Checking
InputData
Unknown
2017-03-18 14:02
InputData
Checking
JobScheduling
Unknown
2017-03-18 14:02
JobScheduling
Checking
JobScheduling
On Hold: after rescheduling 5
2017-03-18 14:02
---
On 18/03/2017 12:33, Simon Fayer wrote:
Hi Giuseppe,
You can generally get more details information about why a job failed from the web interface at:https://dirac.gridpp.ac.uk (Although you have to have your grid certificate installed in your browser to access this).
In this case your jobs have the Application Status of "Input data not available", which as you correctly surmised indicates a problem with the inputfiles.
I've had a look at the JDL which Ganga generated and you have three LFN based files in the sandbox:
"LFN:/gridpp/user/giuseppe.congedo/sim/image_0_details.fits", "LFN:/gridpp/user/giuseppe.congedo/PSF/simssbc_cb2004a_001.fits_0.000_0.804_1.00.fits", "LFN:/gridpp/user/giuseppe.congedo/sim/.lensmc_cache",
If I look in the catalogue (using the dirac-dms-filecatalog-cli tool), it seems like you used a slightly different path for these files when you uploaded them:
/gridpp/user/giuseppe.congedo/sim/sim/image_0_details.fits /gridpp/user/giuseppe.congedo/sim/PSF/simssbc_cb2004a_001.fits_0.000_0.804_1.00.fits /gridpp/user/giuseppe.congedo/sim/.lensmc_cache
(Note the extra "sim/" directory in the first two paths). It should simply be a case of adjusting the paths in your ganga script to match the file catalogue.
It also looks like you may have put these as inputdata as well... There is no need to do this, just specifying them in the inputfiles is sufficient (if you set inputdata, the jobs have to run at a site with those files, whereas inputfiles can be downloaded at any site).
Please let us know if that doesn't work or if you run into any further problems.
Regards, Simon
On Sat, Mar 18, 2017 at 11:21:56AM +0000, Giuseppe Congedo wrote:
Hello all,
I am trying to move my workflow on gridpp, but the problem is, I am a newbie! Andrew Lahiff has been very helpful in giving me good advice along the way - thanks. I hope you too could help me with this.
I set up code, software environment, and data, and it works fine locally on Ganga. To move things on the grid, I put code & environment on cvmfs (thanks Catalin Condurache), and uploaded data on DFC. After submission, I got the error attached below. It looks like the job could not see the input data, and also some output sandbox was not properly initialised.
My submission scripts looks like the example in the userguide. I suspect I might be doing something wrong perhaps here?
j.inputfiles = [ DiracFile('LFN:/gridpp/user/giuseppe.congedo/my_folder/my_file'),
Please note that the job fails no matter if I set DiracLFNBase or not in .gangarc.
I look forward to hearing from you, and thanks so much for your help.
Giuseppe
--- ERROR Unable to finalise job after 36 retries due to error: GangaDiracError: No Output sandbox registered for job 2335379 INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (2 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (3 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (4 of 5) after 2.5-sec delay INFO job 36 status changed to "submitted" INFO job 36 status changed to "running" INFO job 36 status changed to "failed" WARNING An error occured finalising job: 36 WARNING Attemting again (5 of 5) after 2.5-sec delay ERROR GangaDiracError: No Output sandbox registered for job 2335379 ---
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Hi Giuseppe, When using Ganga, the peek stdout function on the website unfortunately shows the wrapper output rather than the main (user) script output. The trick to get around this is to select "Sandbox -> Get Output Sandbox" and have a look in the returned zip file at the Ganga_Executable.log file. In this case the error seems to be: python: can't open file '/cvmfs/scotgrid.gridpp.ac.uk/euclid/gcongedo/.lensmc_env/lib/python2.7/site-packages/lensmc-1.1-py2.7-linux-x86_64.egg/lensmc/bulge_disc/shear_measurement.py': [Errno 2] No such file or directory The file seems to be there in CVMFS, so I suspect the site it ran at (VAC.UKI-NORTHGRID-MAN-HEP.uk) doesn't have that specific CVMFS repo enabled. You could try forcing the job to go to LCG.UKI-SCOTGRID-ECDF.uk[*] (as they almost certainly have the scotgrid repo enabled) and see if it works there. Regards, Simon [*] I don't really know much about Ganga, but I think the syntax for this is something like: j.backend.settings['Destination'] = 'LCG.UKI-SCOTGRID-ECDF.uk' On Sat, Mar 18, 2017 at 03:38:15PM +0000, Giuseppe Congedo wrote:
Hi Simon,
Thanks so much. That helped indeed, the job was correctly dispatched to a site, and data correctly downloaded, which I think it's already good progress!
But it ran into problems afterwards as it crashed a few minutes later. The stdout and stderr don't say that much, so I'm not sure what happened. Also peek in ganga is not very informative, showing only a warning about a non-existent scratch directory. Any clues?
Thanks again for your help.
Giuseppe
================================================================================== Last 8 lines of application output from JobWrapper on 2017-03-18 14:54:47.985686 : CPU Total: 00:00:02 (h:m:s) Normalized CPU Total 24.7 s @ HEP'06 ================================================================================== 2017-03-18 14:54:39 UTC dirac-jobexec INFO: JobID: 2335477 2017-03-18 14:54:39 UTC dirac-jobexec INFO: DIRAC JobID 2335477 is running at site VAC.UKI-NORTHGRID-MAN-HEP.uk Executing StepInstance RunScriptStep1 of type ScriptStep1 ['ScriptStep1'] 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Command is: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: Non-zero status while executing 2: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Output written to Ganga_Executable.log, execution complete. 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: 'exe-script.py' Exited With Status 2 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: ===== Terminating=====
Hi Simon, Many thanks for all your useful tips. Indeed the file exists on CVMFS. Then I tried with j.backend.settings['Destination'] = 'LCG.UKI-SCOTGRID-ECDF.uk' as you suggested, and the job 2335584 went to the site as expected. Good suggestion! Unfortunately it was put on hold because of, again, a failure in the input sandbox, despite the paths were fine and the data stored in the same place, LCG.UKI-SCOTGRID-ECDF.uk. Unexpectedly, though, another identical job, 2335585, I submitted afterwards without the setting above started running on LCG.UKI-LT2-QMUL.uk, which I suppose supports the CVMFS repo. This is officially my first running job on Dirac! I wonder if it's possible to know what sites support the CVMFS repo, so I won't waste jobs, or if it makes sense to change the workflow so I don't use CVMFS any more and avoid these problems. Best wishes Giuseppe On 18/03/2017 16:22, Simon Fayer wrote:
Hi Giuseppe,
When using Ganga, the peek stdout function on the website unfortunately shows the wrapper output rather than the main (user) script output. The trick to get around this is to select "Sandbox -> Get Output Sandbox" and have a look in the returned zip file at the Ganga_Executable.log file. In this case the error seems to be:
python: can't open file '/cvmfs/scotgrid.gridpp.ac.uk/euclid/gcongedo/.lensmc_env/lib/python2.7/site-packages/lensmc-1.1-py2.7-linux-x86_64.egg/lensmc/bulge_disc/shear_measurement.py': [Errno 2] No such file or directory
The file seems to be there in CVMFS, so I suspect the site it ran at (VAC.UKI-NORTHGRID-MAN-HEP.uk) doesn't have that specific CVMFS repo enabled. You could try forcing the job to go to LCG.UKI-SCOTGRID-ECDF.uk[*] (as they almost certainly have the scotgrid repo enabled) and see if it works there.
Regards, Simon
[*] I don't really know much about Ganga, but I think the syntax for this is something like: j.backend.settings['Destination'] = 'LCG.UKI-SCOTGRID-ECDF.uk'
On Sat, Mar 18, 2017 at 03:38:15PM +0000, Giuseppe Congedo wrote:
Hi Simon,
Thanks so much. That helped indeed, the job was correctly dispatched to a site, and data correctly downloaded, which I think it's already good progress!
But it ran into problems afterwards as it crashed a few minutes later. The stdout and stderr don't say that much, so I'm not sure what happened. Also peek in ganga is not very informative, showing only a warning about a non-existent scratch directory. Any clues?
Thanks again for your help.
Giuseppe
================================================================================== Last 8 lines of application output from JobWrapper on 2017-03-18 14:54:47.985686 : CPU Total: 00:00:02 (h:m:s) Normalized CPU Total 24.7 s @ HEP'06 ================================================================================== 2017-03-18 14:54:39 UTC dirac-jobexec INFO: JobID: 2335477 2017-03-18 14:54:39 UTC dirac-jobexec INFO: DIRAC JobID 2335477 is running at site VAC.UKI-NORTHGRID-MAN-HEP.uk Executing StepInstance RunScriptStep1 of type ScriptStep1 ['ScriptStep1'] 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Command is: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: Non-zero status while executing 2: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Output written to Ganga_Executable.log, execution complete. 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: 'exe-script.py' Exited With Status 2 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: ===== Terminating=====
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Hi Giuseppe, There are two problems here: a) there is something odd a ECDF. It fails to load one of the essential (for dirac that is) libraries. Simon and me need to debug this. We might set ECDF offline until we know what is going on b) you are mixing VOs and cvmfs repos. This often works, but tends cause more harm than good. If you are using the gridpp VO for work, you code should go in the gridpp cvmfs repo (which, as it's used much more frequently, tends to be available everywhere). I've c'ed Jeremy and Pete as the gridpp VO admins. I hope they will be able to arrange access for you. Regards, Daniela On 18 March 2017 at 23:04, Giuseppe Congedo <giuseppe.congedo@ed.ac.uk> wrote:
Hi Simon,
Many thanks for all your useful tips. Indeed the file exists on CVMFS. Then I tried with
j.backend.settings['Destination'] = 'LCG.UKI-SCOTGRID-ECDF.uk'
as you suggested, and the job 2335584 went to the site as expected. Good suggestion! Unfortunately it was put on hold because of, again, a failure in the input sandbox, despite the paths were fine and the data stored in the same place, LCG.UKI-SCOTGRID-ECDF.uk.
Unexpectedly, though, another identical job, 2335585, I submitted afterwards without the setting above started running on LCG.UKI-LT2-QMUL.uk, which I suppose supports the CVMFS repo. This is officially my first running job on Dirac!
I wonder if it's possible to know what sites support the CVMFS repo, so I won't waste jobs, or if it makes sense to change the workflow so I don't use CVMFS any more and avoid these problems.
Best wishes Giuseppe
On 18/03/2017 16:22, Simon Fayer wrote:
Hi Giuseppe,
When using Ganga, the peek stdout function on the website unfortunately shows the wrapper output rather than the main (user) script output. The trick to get around this is to select "Sandbox -> Get Output Sandbox" and have a look in the returned zip file at the Ganga_Executable.log file. In this case the error seems to be:
python: can't open file '/cvmfs/scotgrid.gridpp.ac.uk/ euclid/gcongedo/.lensmc_env/lib/python2.7/site-packages/lens mc-1.1-py2.7-linux-x86_64.egg/lensmc/bulge_disc/shear_measurement.py': [Errno 2] No such file or directory
The file seems to be there in CVMFS, so I suspect the site it ran at (VAC.UKI-NORTHGRID-MAN-HEP.uk) doesn't have that specific CVMFS repo enabled. You could try forcing the job to go to LCG.UKI-SCOTGRID-ECDF.uk [*] (as they almost certainly have the scotgrid repo enabled) and see if it works there.
Regards, Simon
[*] I don't really know much about Ganga, but I think the syntax for this is something like: j.backend.settings['Destination'] = 'LCG.UKI-SCOTGRID-ECDF.uk'
On Sat, Mar 18, 2017 at 03:38:15PM +0000, Giuseppe Congedo wrote:
Hi Simon,
Thanks so much. That helped indeed, the job was correctly dispatched to a site, and data correctly downloaded, which I think it's already good progress!
But it ran into problems afterwards as it crashed a few minutes later. The stdout and stderr don't say that much, so I'm not sure what happened. Also peek in ganga is not very informative, showing only a warning about a non-existent scratch directory. Any clues?
Thanks again for your help.
Giuseppe
============================================================ ====================== Last 8 lines of application output from JobWrapper on 2017-03-18 14:54:47.985686 : CPU Total: 00:00:02 (h:m:s) Normalized CPU Total 24.7 s @ HEP'06 ============================================================ ====================== 2017-03-18 14:54:39 UTC dirac-jobexec INFO: JobID: 2335477 2017-03-18 14:54:39 UTC dirac-jobexec INFO: DIRAC JobID 2335477 is running at site VAC.UKI-NORTHGRID-MAN-HEP.uk Executing StepInstance RunScriptStep1 of type ScriptStep1 ['ScriptStep1'] 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Command is: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: Non-zero status while executing 2: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Output written to Ganga_Executable.log, execution complete. 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: 'exe-script.py' Exited With Status 2 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: ===== Terminating=====
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
Hi Daniela, Thanks so much for looking into this. As far as I understand, I would need the software/code to be on cvmfs (so I won't need to send it with the job), and data on DFC. This way, all sites will be able to see both software and data. Please let me know if this looks sensible to you. Thanks again for your help. Giuseppe On 20/03/2017 11:59, Daniela Bauer wrote:
Hi Giuseppe,
There are two problems here: a) there is something odd a ECDF. It fails to load one of the essential (for dirac that is) libraries. Simon and me need to debug this. We might set ECDF offline until we know what is going on
b) you are mixing VOs and cvmfs repos. This often works, but tends cause more harm than good. If you are using the gridpp VO for work, you code should go in the gridpp cvmfs repo (which, as it's used much more frequently, tends to be available everywhere). I've c'ed Jeremy and Pete as the gridpp VO admins. I hope they will be able to arrange access for you.
Regards, Daniela
On 18 March 2017 at 23:04, Giuseppe Congedo <giuseppe.congedo@ed.ac.uk <mailto:giuseppe.congedo@ed.ac.uk>> wrote:
Hi Simon,
Many thanks for all your useful tips. Indeed the file exists on CVMFS. Then I tried with
j.backend.settings['Destination'] = 'LCG.UKI-SCOTGRID-ECDF.uk <http://LCG.UKI-SCOTGRID-ECDF.uk>'
as you suggested, and the job 2335584 went to the site as expected. Good suggestion! Unfortunately it was put on hold because of, again, a failure in the input sandbox, despite the paths were fine and the data stored in the same place, LCG.UKI-SCOTGRID-ECDF.uk <http://LCG.UKI-SCOTGRID-ECDF.uk>.
Unexpectedly, though, another identical job, 2335585, I submitted afterwards without the setting above started running on LCG.UKI-LT2-QMUL.uk <http://LCG.UKI-LT2-QMUL.uk>, which I suppose supports the CVMFS repo. This is officially my first running job on Dirac!
I wonder if it's possible to know what sites support the CVMFS repo, so I won't waste jobs, or if it makes sense to change the workflow so I don't use CVMFS any more and avoid these problems.
Best wishes Giuseppe
On 18/03/2017 16:22, Simon Fayer wrote:
Hi Giuseppe,
When using Ganga, the peek stdout function on the website unfortunately shows the wrapper output rather than the main (user) script output. The trick to get around this is to select "Sandbox -> Get Output Sandbox" and have a look in the returned zip file at the Ganga_Executable.log file. In this case the error seems to be:
python: can't open file '/cvmfs/scotgrid.gridpp.ac.uk/euclid/gcongedo/.lensmc_env/lib/python2.7/site-packages/lensmc-1.1-py2.7-linux-x86_64.egg/lensmc/bulge_disc/shear_measurement.py <http://scotgrid.gridpp.ac.uk/euclid/gcongedo/.lensmc_env/lib/python2.7/site-packages/lensmc-1.1-py2.7-linux-x86_64.egg/lensmc/bulge_disc/shear_measurement.py>': [Errno 2] No such file or directory
The file seems to be there in CVMFS, so I suspect the site it ran at (VAC.UKI-NORTHGRID-MAN-HEP.uk <http://VAC.UKI-NORTHGRID-MAN-HEP.uk>) doesn't have that specific CVMFS repo enabled. You could try forcing the job to go to LCG.UKI-SCOTGRID-ECDF.uk <http://LCG.UKI-SCOTGRID-ECDF.uk>[*] (as they almost certainly have the scotgrid repo enabled) and see if it works there.
Regards, Simon
[*] I don't really know much about Ganga, but I think the syntax for this is something like: j.backend.settings['Destination'] = 'LCG.UKI-SCOTGRID-ECDF.uk <http://LCG.UKI-SCOTGRID-ECDF.uk>'
On Sat, Mar 18, 2017 at 03:38:15PM +0000, Giuseppe Congedo wrote:
Hi Simon,
Thanks so much. That helped indeed, the job was correctly dispatched to a site, and data correctly downloaded, which I think it's already good progress!
But it ran into problems afterwards as it crashed a few minutes later. The stdout and stderr don't say that much, so I'm not sure what happened. Also peek in ganga is not very informative, showing only a warning about a non-existent scratch directory. Any clues?
Thanks again for your help.
Giuseppe
================================================================================== Last 8 lines of application output from JobWrapper on 2017-03-18 14:54:47.985686 : CPU Total: 00:00:02 (h:m:s) Normalized CPU Total 24.7 s @ HEP'06 ================================================================================== 2017-03-18 14:54:39 UTC dirac-jobexec INFO: JobID: 2335477 2017-03-18 14:54:39 UTC dirac-jobexec INFO: DIRAC JobID 2335477 is running at site VAC.UKI-NORTHGRID-MAN-HEP.uk <http://VAC.UKI-NORTHGRID-MAN-HEP.uk> Executing StepInstance RunScriptStep1 of type ScriptStep1 ['ScriptStep1'] 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Command is: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: Non-zero status while executing 2: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Output written to Ganga_Executable.log, execution complete. 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: 'exe-script.py' Exited With Status 2 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: ===== Terminating=====
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users>
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk <mailto:daniela.bauer@imperial.ac.uk> HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/ <http://www.hep.ph.ic.ac.uk/%7Edbauer/>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Hi Giuseppe, in a word, yes. If you don't hear from Pete/Jeremy about cvmfs access, please let me know (off list). Regards, Daniela On 20 March 2017 at 14:08, Giuseppe Congedo <giuseppe.congedo@ed.ac.uk> wrote:
Hi Daniela,
Thanks so much for looking into this. As far as I understand, I would need the software/code to be on cvmfs (so I won't need to send it with the job), and data on DFC. This way, all sites will be able to see both software and data. Please let me know if this looks sensible to you.
Thanks again for your help.
Giuseppe
On 20/03/2017 11:59, Daniela Bauer wrote:
Hi Giuseppe,
There are two problems here: a) there is something odd a ECDF. It fails to load one of the essential (for dirac that is) libraries. Simon and me need to debug this. We might set ECDF offline until we know what is going on
b) you are mixing VOs and cvmfs repos. This often works, but tends cause more harm than good. If you are using the gridpp VO for work, you code should go in the gridpp cvmfs repo (which, as it's used much more frequently, tends to be available everywhere). I've c'ed Jeremy and Pete as the gridpp VO admins. I hope they will be able to arrange access for you.
Regards, Daniela
On 18 March 2017 at 23:04, Giuseppe Congedo <giuseppe.congedo@ed.ac.uk> wrote:
Hi Simon,
Many thanks for all your useful tips. Indeed the file exists on CVMFS. Then I tried with
j.backend.settings['Destination'] = 'LCG.UKI-SCOTGRID-ECDF.uk'
as you suggested, and the job 2335584 went to the site as expected. Good suggestion! Unfortunately it was put on hold because of, again, a failure in the input sandbox, despite the paths were fine and the data stored in the same place, LCG.UKI-SCOTGRID-ECDF.uk.
Unexpectedly, though, another identical job, 2335585, I submitted afterwards without the setting above started running on LCG.UKI-LT2-QMUL.uk, which I suppose supports the CVMFS repo. This is officially my first running job on Dirac!
I wonder if it's possible to know what sites support the CVMFS repo, so I won't waste jobs, or if it makes sense to change the workflow so I don't use CVMFS any more and avoid these problems.
Best wishes Giuseppe
On 18/03/2017 16:22, Simon Fayer wrote:
Hi Giuseppe,
When using Ganga, the peek stdout function on the website unfortunately shows the wrapper output rather than the main (user) script output. The trick to get around this is to select "Sandbox -> Get Output Sandbox" and have a look in the returned zip file at the Ganga_Executable.log file. In this case the error seems to be:
python: can't open file '/cvmfs/scotgrid.gridpp.ac.uk/ euclid/gcongedo/.lensmc_env/lib/python2.7/site-packages/lens mc-1.1-py2.7-linux-x86_64.egg/lensmc/bulge_disc/shear_measurement.py': [Errno 2] No such file or directory
The file seems to be there in CVMFS, so I suspect the site it ran at (VAC.UKI-NORTHGRID-MAN-HEP.uk) doesn't have that specific CVMFS repo enabled. You could try forcing the job to go to LCG.UKI-SCOTGRID-ECDF.uk [*] (as they almost certainly have the scotgrid repo enabled) and see if it works there.
Regards, Simon
[*] I don't really know much about Ganga, but I think the syntax for this is something like: j.backend.settings['Destination'] = 'LCG.UKI-SCOTGRID-ECDF.uk'
On Sat, Mar 18, 2017 at 03:38:15PM +0000, Giuseppe Congedo wrote:
Hi Simon,
Thanks so much. That helped indeed, the job was correctly dispatched to a site, and data correctly downloaded, which I think it's already good progress!
But it ran into problems afterwards as it crashed a few minutes later. The stdout and stderr don't say that much, so I'm not sure what happened. Also peek in ganga is not very informative, showing only a warning about a non-existent scratch directory. Any clues?
Thanks again for your help.
Giuseppe
============================================================ ====================== Last 8 lines of application output from JobWrapper on 2017-03-18 14:54:47.985686 : CPU Total: 00:00:02 (h:m:s) Normalized CPU Total 24.7 s @ HEP'06 ============================================================ ====================== 2017-03-18 14:54:39 UTC dirac-jobexec INFO: JobID: 2335477 2017-03-18 14:54:39 UTC dirac-jobexec INFO: DIRAC JobID 2335477 is running at site VAC.UKI-NORTHGRID-MAN-HEP.uk Executing StepInstance RunScriptStep1 of type ScriptStep1 ['ScriptStep1'] 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Command is: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: Non-zero status while executing 2: /scratch/plt00/2335477/exe-script.py 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: Output written to Ganga_Executable.log, execution complete. 2017-03-18 14:54:40 UTC dirac-jobexec/Script ERROR: 'exe-script.py' Exited With Status 2 2017-03-18 14:54:40 UTC dirac-jobexec/Script INFO: ===== Terminating=====
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 <+44%2020%207594%207810> http://www.hep.ph.ic.ac.uk/~dbauer/
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
participants (4)
-
Daniela Bauer
-
Giuseppe Congedo
-
Simon Fayer
-
Will Furnell