- Gridpp-Dirac-Users - imperial.ac.uk

ERROR Error initializing tool 'Generation.Special.ReadLHEfileProduction'
by Arka Santra 23 Jul '18

23 Jul '18

Hello grid experts, I am trying to submit a few jobs to Dirac grid. But unfortunately, all of my jobs are failing. I do not understand the reason. I am attaching the config file, bash file and the output log of the failed run. The particular error is this one: "Generation.Special INFO Generating Special events. F77Utils::openOld ERROR Error in opening LUN=97 FILE='unweighted_events.lhe' Generation.Special.Re... ERROR ReadLHEfileProduction:: Could not open the file 'unweighted_events.lhe' StatusCode=FAILURE ToolSvc ERROR Error initializing tool 'Generation.Special.ReadLHEfileProduction' Generation.Special FATAL Special:: Exception throw: tool():: Could not retrieve Tool 'ReadLHEfileProduction' StatusCode=FAILURE Generation.Special.sy... FATAL Exception with tag= is caught Generation.Special.sy... ERROR Generation.Special:: tool():: Could not retrieve Tool 'ReadLHEfileProduction' StatusCode=FAILURE ToolSvc ERROR Error initializing tool 'Generation.Special' Generation FATAL Generation:: Exception throw: tool():: Could not retrieve Tool 'Special' StatusCode=FAILURE Generation FATAL Exception with tag= is caught Generation ERROR Generation:: tool():: Could not retrieve Tool 'Special' StatusCode=FAILURE GeneratorSlotMainSeq ERROR GaudiSequencer:: Can not initialize Generation StatusCode=FAILURE Generator ERROR GaudiSequencer:: Can not initialize GeneratorSlotMainSeq StatusCode=FAILURE GaussSequencer ERROR GaudiSequencer:: Can not initialize Generator StatusCode=FAILURE EventLoopMgr ERROR Unable to initialize Algorithm: GaussSequencer ServiceManager ERROR Unable to initialize Service: EventLoopMgr " For some reason, Dirac could not retrieve Tool 'ReadLHEfileProduction' . What should I do here? I am sorry if this is not Dirac problem, but the problem with Ganga. If that is the case, may I know how I solve this Ganga problem? I am also attaching my gangarc file and Ganga Executable log. Thank you very much, Arka -- =============================================================== *Arka Santra* *Post Doctoral Researcher* * Instituto de Fisica Corpuscular* * Parque Cientifico* * C/Catedratico Jose Beltran, 2* * E-46980, Paterna* * Espana* ===============================================================

2 1

Re: [Gridpp-Dirac-Users] FW: issue - idle jobs on GRIDPP
by Arka Santra 18 Jul '18

18 Jul '18

Hi Daniela, I put this line to my gangarc file as suggested by Rob: [Registry] DisableLoadCheck=False and resubmitted all my jobs on July 4, 2018. But again, I see all my jobs are in the 'new' state even after 12 days of submitting: https://docs.google.com/document/d/19YG32Fb47NWeykPS1iFww4Zh0JC43G0E0JL3uDC… As the case was before, I do not see any job in the Dirac JobMonitor. I will be really grateful if I get some suggestions on how to solve this grid/Ganga issue. Thank you very much, Arka On Wed, Jul 4, 2018 at 8:56 AM Daniel Felea <Daniel.Felea(a)cern.ch> wrote: > Hi Daniela, > > Thank you very much ! > > Hopefully, changing the settings in Arka's .gangarc : > > [Registry] > DisableLoadCheck=False > > as recommended by Rob will solve, yet again, this issue. > > Thanks again and cheers, > Daniel > > ________________________________________ > From: gridpp-dirac-users-bounces(a)imperial.ac.uk [ > gridpp-dirac-users-bounces(a)imperial.ac.uk] on behalf of Daniela Bauer [ > daniela.bauer.grid(a)googlemail.com] > Sent: 03 July 2018 16:16 > To: gridpp-dirac-users(a)imperial.ac.uk > Cc: Santra.Arka(a)ific.uv.es; santra.arka(a)gmail.com > Subject: Re: [Gridpp-Dirac-Users] FW: issue - idle jobs on GRIDPP > > Hi Daniel, > > I though there was another thread where we all agreed this was a Ganga > issue, rather than DIRAC. Arka has no jobs in the Waiting state on the > DIRAC server. > You can always check this for yourself by going to: > https://dirac.gridpp.ac.uk:8443/DIRAC > clicking on JobMonitor and fill out the appropriate selectors. > > Regards, > Daniela > > > On 3 July 2018 at 12:53, Daniel Felea <Daniel.Felea(a)cern.ch> wrote: > > > > Hello All, > > > > Would someone please take a look into the issue described below. Arka has > > lots of MoEDAL jobs stuck on Grid since the 3rd of June. > > > > Thank you very much in advance for your help ! > > > > Thanks and cheers, > > Daniel > > > > > > ________________________________ > > > > On Sun, Jun 24, 2018 at 10:04 AM Daniel Felea <Daniel.Felea(a)cern.ch> > wrote: > >> > >> > >> ________________________________ > >> From: Arka Santra [santra.arka(a)gmail.com] > >> Sent: 24 June 2018 09:00 > >> To: Daniel Felea > >> Subject: Re: issue - idle jobs on GRIDPP > >> > >> Hi Daniel, > >> > >> ...I submitted 1080 jobs on June 3, 2018. This is the status of all the > >> jobs as of now: > >> > >> > https://docs.google.com/document/d/1o6zYFmICCcyVsbPDc7z3DmRkkLz413SFFsyQgSq… > >> You can see, most of my jobs are in the submitted state (jobs 1494 and > >> more). Only first couple of hundreds of jobs were done by the grid. They > >> were done on June 3 itself. Nothing was processed after that.... > >> > >> ...Even today, after exactly a month, all my jobs remain in the same > >> state. No progress at all! I think this is a grid issue, otherwise, 30% > of > >> my jobs would not be processed. They would also be in stalled mode. > >> > >> Thank you, > >> Arka > >> > > > > -- > > _______________________________________________ > > Gridpp-Dirac-Users mailing list > > Gridpp-Dirac-Users(a)imperial.ac.uk > > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users > > > > > > -- > Sent from the pit of despair > > ----------------------------------------------------------- > daniela.bauer(a)imperial.ac.uk > HEP Group/Physics Dep > Imperial College > London, SW7 2BW > Tel: +44-(0)20-75947810 > http://www.hep.ph.ic.ac.uk/~dbauer/ > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users > -- =============================================================== *Arka Santra* *Post Doctoral Researcher* * Instituto de Fisica Corpuscular* * Parque Cientifico* * C/Catedratico Jose Beltran, 2* * E-46980, Paterna* * Espana* ===============================================================

3 2

FW: issue - idle jobs on GRIDPP
by Daniel Felea 03 Jul '18

03 Jul '18

Hello All, Would someone please take a look into the issue described below. Arka has lots of MoEDAL jobs stuck on Grid since the 3rd of June. Thank you very much in advance for your help ! Thanks and cheers, Daniel ________________________________ On Sun, Jun 24, 2018 at 10:04 AM Daniel Felea <Daniel.Felea(a)cern.ch<mailto:Daniel.Felea@cern.ch>> wrote: ________________________________ From: Arka Santra [santra.arka(a)gmail.com<mailto:santra.arka@gmail.com>] Sent: 24 June 2018 09:00 To: Daniel Felea Subject: Re: issue - idle jobs on GRIDPP Hi Daniel, ...I submitted 1080 jobs on June 3, 2018. This is the status of all the jobs as of now: https://docs.google.com/document/d/1o6zYFmICCcyVsbPDc7z3DmRkkLz413SFFsyQgSq… You can see, most of my jobs are in the submitted state (jobs 1494 and more). Only first couple of hundreds of jobs were done by the grid. They were done on June 3 itself. Nothing was processed after that.... ...Even today, after exactly a month, all my jobs remain in the same state. No progress at all! I think this is a grid issue, otherwise, 30% of my jobs would not be processed. They would also be in stalled mode. Thank you, Arka

2 1

jobs in submitted state for more than a week
by Arka Santra 10 May '18

10 May '18

Hi, I am trying to do a MoEDAL simulation with Ganga. I submitted 1080 jobs initially. While 900 jobs were processed correctly producing the desired output, rest of the 180 jobs were stuck in the grid (with the status 'running') for more than 3 weeks. Last week, I killed those jobs and resent them to the gird. Since then, all my jobs are in the submitted state. Not a single one has started to run. This is the script used to submit the jobs: https://github.com/asantra/ScriptsForDiracGrid/blob/master/make_dirac_lhe_j… This is the bash script: https://github.com/asantra/ScriptsForDiracGrid/blob/master/run_lhe_v49r8.sh Can you look into this issue? Please let me know if you need more details to investigate the issue. Thank you, Arka -- =============================================================== *Arka Santra* *Post Doctoral Researcher* * Instituto de Fisica Corpuscular* * Parque Cientifico* * C/Catedratico Jose Beltran, 2* * E-46980, Paterna* * Espana* ===============================================================

3 2

Re: [Gridpp-Dirac-Users] Transient errors with DIRAC jobs
by Daniela Bauer 24 Apr '18

24 Apr '18

Hi Rohini, Please always include the mailing list. While Simon and me administer the DIRAC instance we don't actually use it and other people might be better placed to answer your questions. Specifically Job IDs 8920431, 8920312, 8907180, 8897518 are failing with > Input data errors. However we have confirmed that the input data does in > fact exist and is accessible (locally with dirac-dms-get-file) > > This looks like a catalogue error. Unfortunately when I try and search the logs for the first job I find: runit/WorkloadManagement/Optimizers_1/log/@400000005ad9a5a434a9aaa4.s:2018-04-19 10:37:53 UTC WorkloadManagement/Optimizers_1/WorkloadManagement/JobScheduling INFO: [JID 8920431] Single chosen site LCG.UKI-NORTHGRID-MAN-HEP.uk specified runit/WorkloadManagement/Optimizers_1/log/@400000005ad9a5a434a9aaa4.s:2018-04-19 10:37:53 UTC WorkloadManagement/Optimizers_1/WorkloadManagement/JobScheduling INFO: [JID 8920431] Site candidates are ['CLOUD.Datacentred.uk', ' VAC.UKI-LT2-UCL-HEP.uk', 'VAC.UKI-NORTHGRID-MAN-HEP.uk', ' LCG.UKI-NORTHGRID-MAN-HEP.uk'] runit/WorkloadManagement/Optimizers_1/log/@400000005ad9a5a434a9aaa4.s:2018-04-19 10:37:53 UTC WorkloadManagement/Optimizers_1/WorkloadManagement/JobScheduling INFO: [JID 8920431] No staging required runit/WorkloadManagement/Optimizers_1/log/@400000005ad9a5a434a9aaa4.s:2018-04-19 10:37:53 UTC WorkloadManagement/Optimizers_1/WorkloadManagement/JobScheduling INFO: [JID 8920431] Only site LCG.UKI-NORTHGRID-MAN-HEP.uk is candidate runit/WorkloadManagement/Optimizers_1/log/@400000005ad9a5a434a9aaa4.s:2018-04-19 10:37:53 UTC WorkloadManagement/Optimizers_1/WorkloadManagement/JobScheduling INFO: [JID 8920431] Done which as you can see has no error, so I have nothing to go on. I really don't know what to do about this one, I will go and forward it to the DIRAC developers. (later it says: runit/WorkloadManagement/Optimizers_1/log/@400000005ad9a5a434a9aaa4.s:2018-04-19 11:30:33 UTC WorkloadManagement/Optimizers_1/WorkloadManagement/JobScheduling INFO: [JID 8920431] Not in checking state. Avoid fast track but even that is not an error) Does the error above disappear when you rerun the jobs ? > Also, from time to time I have seen jobs fail with ApplicationStatus > 'Cannot retrieve banned sites from JobDB' (most recently Job ID 8897033) > and also 'FileCatalog error ( 1604 : Failed to perform getReplicas from any > catalog)' Job ID 8897076 (several from job group rohini.joshi.20180418103426) > and Therese has seen this problem too with Job ID 8865042 > These errors seem to be transient and at times re-running jobs resolves > the problem. > We assume this is a bug in DIRAC as this has come up for other DIRAC instances as well. We've done various modifications to our DIRAC instance (mainly more of everything, as it looks a bit like a load/access problem), but we cannot reproduce it on command, which makes debugging very hard. We'll keep looking. > Just for some context, my jobs are uploading some data to RAL (in a lazy > way) and are essentially just running gfal-copy command to upload data from > DIRAC storage at Manchester to RAL. Therese's job is trying to run a > singularity container on a Manchester GPU node. > Do you have a retry loop (with a sleep between retries) for your uploads ? @Therese: How do you target the GPU queue ? Sorry that I can't be more helpful at the moment. Regards, Daniela -- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer(a)imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/

1 2

jobs failing with the error message: 'Transport endpoint is not connected'
by Arka Santra 17 Apr '18

17 Apr '18

Hello, I am trying to submit jobs to the grid. My jobs are failing with the error message: "Transport endpoint is not connected" Sometimes, when I resubmit the same job, I see that the job is processed perfectly. This confuses me more. I have attached the script that I used to submit the job. I have also attached the std.out and Ganga_Executable.log for your perusal. I will be grateful if you help me to debug this issue. Thank you, Arka -- =============================================================== *Arka Santra* *Post Doctoral Researcher* * Instituto de Fisica Corpuscular* * Parque Cientifico* * C/Catedratico Jose Beltran, 2* * E-46980, Paterna* * Espana* ===============================================================

3 2

Dirac Server Web interface Error
by Matt Hogan 14 Apr '18

14 Apr '18

Hello DIRAC experts, I tried to use the web interface hosted at Imperial College https://dirac.gridpp.ac.uk/ and I get a series of errors indicating some PEM file is now invalid. You can see what I get in the attachments below. I cannot use the Job Monitor anymore which is really a nice feature. Can someone look into this problem? Cheers, Matt

2 2

Installation Failing Due to LCG Bundle
by Matt Hogan 07 Apr '18

07 Apr '18

Hello DIRAC users, I am trying to install the client using the following settings recommended by my experiment ./dirac-install -r v6r19p2 -g v13r0 And I run into a error because of the LCG bundle failed to download. The specific tarball it says it needs doesn't exist and I confirmed it does not exist when I check the DIRAC host pages. I attached what I got running the installation. I would try to patch the scripts myself, but I cannot easily find the problematic lines using grep. In the meantime, I disabled the bundle installation. Can I get some help how to make the bundle install work? Thanks, Matt

2 2

Error with grid jobs
by Arka Santra 04 Apr '18

04 Apr '18

Hello, I wanted to send few jobs to the Ganga. This is the script I wrote: https://github.com/asantra/ScriptsForDiracGrid/blob/master/make_dirac_lhe_j… But the jobs failed. I have attached the stdout and log file of one of the jobs. How do I get rid of the issue? I resubmitted one job, but the job seems to be stuck with the grid. Is there any problem with the grid currently? Thanks, Arka -- =============================================================== *Arka Santra* *Post Doctoral Researcher* * Instituto de Fisica Corpuscular* * Parque Cientifico* * C/Catedratico Jose Beltran, 2* * E-46980, Paterna* * Espana* ===============================================================

2 1

Can't contact DIRAC CS
by Arka Santra 27 Mar '18

27 Mar '18

Hi, I wanted to submit a grid job on Ganga. Even though I can generate the proxy locally using this command: dirac-proxy-init -g vo.moedal.org_user -M, but when on the grid, I am getting this error: Ganga In [1]: execfile('make_dirac_lhe_job.py') * * Generating the MoEDAL simulation (LHE) configuration file for Ganga (DIRAC) * * * Configuration filename: 'cfg_13TeV_PhotonFusion_myRun_betaIndependent_SpinZero_q10_run_05_default' INFO submitting job 50 INFO job 50 status changed to "submitting" INFO Preparing Executable application. INFO Created shared directory: conf-2958b76e-93ea-4f5c-a338-5a99cef6564a INFO Preparing subjobs Generating proxy... Enter Certificate password: Can't contact DIRAC CS: Reason(s): Unauthorized query ( 1111 : Unauthorized query) (User possibly not registered with dirac server) It seems I am not registered with the dirac server. May I know how I can do that and use the grid to submit jobs? Thanks, Arka -- =============================================================== *Arka Santra* *Post Doctoral Researcher* * Instituto de Fisica Corpuscular* * Parque Cientifico* * C/Catedratico Jose Beltran, 2* * E-46980, Paterna* * Espana* ===============================================================

4 18