Re: [Gridpp-Dirac-Users] FW: issue - idle jobs on GRIDPP
Hi Daniela, I put this line to my gangarc file as suggested by Rob: [Registry] DisableLoadCheck=False and resubmitted all my jobs on July 4, 2018. But again, I see all my jobs are in the 'new' state even after 12 days of submitting: https://docs.google.com/document/d/19YG32Fb47NWeykPS1iFww4Zh0JC43G0E0JL3uDC5... As the case was before, I do not see any job in the Dirac JobMonitor. I will be really grateful if I get some suggestions on how to solve this grid/Ganga issue. Thank you very much, Arka On Wed, Jul 4, 2018 at 8:56 AM Daniel Felea <Daniel.Felea@cern.ch> wrote:
Hi Daniela,
Thank you very much !
Hopefully, changing the settings in Arka's .gangarc :
[Registry] DisableLoadCheck=False
as recommended by Rob will solve, yet again, this issue.
Thanks again and cheers, Daniel
________________________________________ From: gridpp-dirac-users-bounces@imperial.ac.uk [ gridpp-dirac-users-bounces@imperial.ac.uk] on behalf of Daniela Bauer [ daniela.bauer.grid@googlemail.com] Sent: 03 July 2018 16:16 To: gridpp-dirac-users@imperial.ac.uk Cc: Santra.Arka@ific.uv.es; santra.arka@gmail.com Subject: Re: [Gridpp-Dirac-Users] FW: issue - idle jobs on GRIDPP
Hi Daniel,
I though there was another thread where we all agreed this was a Ganga issue, rather than DIRAC. Arka has no jobs in the Waiting state on the DIRAC server. You can always check this for yourself by going to: https://dirac.gridpp.ac.uk:8443/DIRAC clicking on JobMonitor and fill out the appropriate selectors.
Regards, Daniela
On 3 July 2018 at 12:53, Daniel Felea <Daniel.Felea@cern.ch> wrote:
Hello All,
Would someone please take a look into the issue described below. Arka has lots of MoEDAL jobs stuck on Grid since the 3rd of June.
Thank you very much in advance for your help !
Thanks and cheers, Daniel
________________________________
On Sun, Jun 24, 2018 at 10:04 AM Daniel Felea <Daniel.Felea@cern.ch>
wrote:
________________________________ From: Arka Santra [santra.arka@gmail.com] Sent: 24 June 2018 09:00 To: Daniel Felea Subject: Re: issue - idle jobs on GRIDPP
Hi Daniel,
...I submitted 1080 jobs on June 3, 2018. This is the status of all the jobs as of now:
https://docs.google.com/document/d/1o6zYFmICCcyVsbPDc7z3DmRkkLz413SFFsyQgSqI...
You can see, most of my jobs are in the submitted state (jobs 1494 and more). Only first couple of hundreds of jobs were done by the grid. They were done on June 3 itself. Nothing was processed after that....
...Even today, after exactly a month, all my jobs remain in the same state. No progress at all! I think this is a grid issue, otherwise, 30% of my jobs would not be processed. They would also be in stalled mode.
Thank you, Arka
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- =============================================================== *Arka Santra* *Post Doctoral Researcher* * Instituto de Fisica Corpuscular* * Parque Cientifico* * C/Catedratico Jose Beltran, 2* * E-46980, Paterna* * Espana* ===============================================================
Hi Arka, Unfortunately it appears that all of your jobs are in the completed state in DIRAC again, so I suspect it's still a ganga problem and changing the gangarc hasn't solved it. I think you'll have to contact the ganga developers again and ask for further help with debugging the issue. (I think Rob is on this list, so hopefully he'll spot this e-mail!). If you open the job monitor, try setting the "Owner" field to your name (plus csic.es) using the drop-down and then press submit at the bottom and you should see all of your jobs. Regards, Simon On Mon, Jul 16, 2018 at 07:54:51PM +0200, Arka Santra wrote:
Hi Daniela, I put this line to my gangarc file as suggested by Rob:
[Registry] DisableLoadCheck=False
and resubmitted all my jobs on July 4, 2018. But again, I see all my jobs are in the 'new' state even after 12 days of submitting: https://docs.google.com/document/d/19YG32Fb47NWeykPS1iFww4Zh0JC43G0E0JL3uDC5...
As the case was before, I do not see any job in the Dirac JobMonitor.
I will be really grateful if I get some suggestions on how to solve this grid/Ganga issue.
Thank you very much, Arka
On Wed, Jul 4, 2018 at 8:56 AM Daniel Felea <Daniel.Felea@cern.ch> wrote:
Hi Daniela,
Thank you very much !
Hopefully, changing the settings in Arka's .gangarc :
[Registry] DisableLoadCheck=False
as recommended by Rob will solve, yet again, this issue.
Thanks again and cheers, Daniel
________________________________________ From: gridpp-dirac-users-bounces@imperial.ac.uk [ gridpp-dirac-users-bounces@imperial.ac.uk] on behalf of Daniela Bauer [ daniela.bauer.grid@googlemail.com] Sent: 03 July 2018 16:16 To: gridpp-dirac-users@imperial.ac.uk Cc: Santra.Arka@ific.uv.es; santra.arka@gmail.com Subject: Re: [Gridpp-Dirac-Users] FW: issue - idle jobs on GRIDPP
Hi Daniel,
I though there was another thread where we all agreed this was a Ganga issue, rather than DIRAC. Arka has no jobs in the Waiting state on the DIRAC server. You can always check this for yourself by going to: https://dirac.gridpp.ac.uk:8443/DIRAC clicking on JobMonitor and fill out the appropriate selectors.
Regards, Daniela
On 3 July 2018 at 12:53, Daniel Felea <Daniel.Felea@cern.ch> wrote:
Hello All,
Would someone please take a look into the issue described below. Arka has lots of MoEDAL jobs stuck on Grid since the 3rd of June.
Thank you very much in advance for your help !
Thanks and cheers, Daniel
________________________________
On Sun, Jun 24, 2018 at 10:04 AM Daniel Felea <Daniel.Felea@cern.ch>
wrote:
________________________________ From: Arka Santra [santra.arka@gmail.com] Sent: 24 June 2018 09:00 To: Daniel Felea Subject: Re: issue - idle jobs on GRIDPP
Hi Daniel,
...I submitted 1080 jobs on June 3, 2018. This is the status of all the jobs as of now:
https://docs.google.com/document/d/1o6zYFmICCcyVsbPDc7z3DmRkkLz413SFFsyQgSqI...
You can see, most of my jobs are in the submitted state (jobs 1494 and more). Only first couple of hundreds of jobs were done by the grid. They were done on June 3 itself. Nothing was processed after that....
...Even today, after exactly a month, all my jobs remain in the same state. No progress at all! I think this is a grid issue, otherwise, 30% of my jobs would not be processed. They would also be in stalled mode.
Thank you, Arka
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Hi, Yes, apologies from myself I was afk for a few days. This sounds like your Ganga session has not been terminated correctly at some stage in the past. If the job is marked as 'new' to Ganga, then you will need to submit it again ( j.submit() ) as the information linking the job inside Ganga and the job inside DIRAC has been lost. ** A word of warning is that Ganga flushes all of it's database info when exiting due to some performance and design requirements. If Ganga fails to close correctly (lost connection to the machine where it's running) or is forcibly closed (hitting Ctrl+C) then the job data _will_ become corrupt and you will likely hit situations such as this. Ganga is unable to recover jobs in a state of 'new' or 'submitting' on disk as this is effectively the same state. 'submitted', 'completing', 'failed' and 'completed' jobs may have useful information in them and Ganga is able to correctly get the output of these jobs and place it in the appropriate local directories. Hope this is of some help, Rob ** It is possible to recover this but it is entirely a manual, detailed, painful process so is ill advised to attempt this. On 16/07/2018 07:40 PM, Simon Fayer wrote:
Hi Arka,
Unfortunately it appears that all of your jobs are in the completed state in DIRAC again, so I suspect it's still a ganga problem and changing the gangarc hasn't solved it.
I think you'll have to contact the ganga developers again and ask for further help with debugging the issue. (I think Rob is on this list, so hopefully he'll spot this e-mail!).
If you open the job monitor, try setting the "Owner" field to your name (plus csic.es) using the drop-down and then press submit at the bottom and you should see all of your jobs.
Regards, Simon
On Mon, Jul 16, 2018 at 07:54:51PM +0200, Arka Santra wrote:
Hi Daniela, I put this line to my gangarc file as suggested by Rob:
[Registry] DisableLoadCheck=False
and resubmitted all my jobs on July 4, 2018. But again, I see all my jobs are in the 'new' state even after 12 days of submitting: https://docs.google.com/document/d/19YG32Fb47NWeykPS1iFww4Zh0JC43G0E0JL3uDC5...
As the case was before, I do not see any job in the Dirac JobMonitor.
I will be really grateful if I get some suggestions on how to solve this grid/Ganga issue.
Thank you very much, Arka
On Wed, Jul 4, 2018 at 8:56 AM Daniel Felea <Daniel.Felea@cern.ch> wrote:
Hi Daniela,
Thank you very much !
Hopefully, changing the settings in Arka's .gangarc :
[Registry] DisableLoadCheck=False
as recommended by Rob will solve, yet again, this issue.
Thanks again and cheers, Daniel
________________________________________ From: gridpp-dirac-users-bounces@imperial.ac.uk [ gridpp-dirac-users-bounces@imperial.ac.uk] on behalf of Daniela Bauer [ daniela.bauer.grid@googlemail.com] Sent: 03 July 2018 16:16 To: gridpp-dirac-users@imperial.ac.uk Cc: Santra.Arka@ific.uv.es; santra.arka@gmail.com Subject: Re: [Gridpp-Dirac-Users] FW: issue - idle jobs on GRIDPP
Hi Daniel,
I though there was another thread where we all agreed this was a Ganga issue, rather than DIRAC. Arka has no jobs in the Waiting state on the DIRAC server. You can always check this for yourself by going to: https://dirac.gridpp.ac.uk:8443/DIRAC clicking on JobMonitor and fill out the appropriate selectors.
Regards, Daniela
On 3 July 2018 at 12:53, Daniel Felea <Daniel.Felea@cern.ch> wrote:
Hello All,
Would someone please take a look into the issue described below. Arka has lots of MoEDAL jobs stuck on Grid since the 3rd of June.
Thank you very much in advance for your help !
Thanks and cheers, Daniel
________________________________
On Sun, Jun 24, 2018 at 10:04 AM Daniel Felea <Daniel.Felea@cern.ch>
wrote:
________________________________ From: Arka Santra [santra.arka@gmail.com] Sent: 24 June 2018 09:00 To: Daniel Felea Subject: Re: issue - idle jobs on GRIDPP
Hi Daniel,
...I submitted 1080 jobs on June 3, 2018. This is the status of all the jobs as of now:
https://docs.google.com/document/d/1o6zYFmICCcyVsbPDc7z3DmRkkLz413SFFsyQgSqI...
You can see, most of my jobs are in the submitted state (jobs 1494 and more). Only first couple of hundreds of jobs were done by the grid. They were done on June 3 itself. Nothing was processed after that....
...Even today, after exactly a month, all my jobs remain in the same state. No progress at all! I think this is a grid issue, otherwise, 30% of my jobs would not be processed. They would also be in stalled mode.
Thank you, Arka
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
participants (3)
-
Arka Santra
-
Robert Currie
-
Simon Fayer