Hi Sam, Daniela Thanks for the response. "Does that file exist in other instances of the LSST CVMFS repo?” - I’m not sure what you mean by that? "Is the CVMFS repo for LSST available at the site the job is running at?” - I have no idea how to check. I’m still a bit of a newbie when it comes to GridPP. I was given a workflow that used to work, but now doesn’t, and I’ve been using it to help understand how GridPP works. Given the issues, it actually has, but it’s not been too clear at times how to do certain things. I was told by our local GridPP person that the sites that accept jobs from LSST VOs should have the LSST CVMFS repo available. The only reference I could find that details how to connect to an LSST repo is at https://github.com/airnandez/lsst-cvmfs <https://github.com/airnandez/lsst-cvmfs>, which is what I used to test the workflow locally before uploading to the grid (which is where this thread started, when uploading the new workflow that did work with the lsst.in2ps.fr CVMFS repo). I can’t find any info on how to connect my CernVM to the lsst.opensciencegrid.org in order to know what path to source to enable the workflow to use EUPS. Any information would be greatly appreciated. Cheers Darren ------------------------------------------------------------------------- Dr Darren J. White Applications Developer EPCC, University of Edinburgh Email: d.white@epcc.ed.ac.uk Telephone: +44 (0) 131 6513415 Room 2411, James Clerk Maxwell Building, Kings Buildings, Mayfield Road, Edinburgh EH9 3JZ, United Kingdom -------------------------------------------------------------------------
On 4 Apr 2017, at 12:04, Sam Skipsey <sskipsey@googlemail.com> wrote:
Strictly, the LSST CVMFS repo will be available for jobs submitted as any user (it's not a protected filesystem), as long as the repo is enabled at the site. Does that file exist in other instances of the LSST CVMFS repo? Is the CVMFS repo for LSST available at the site the job is running at?
Sam
On Tue, 4 Apr 2017 at 11:56 Darren White <d.white@epcc.ed.ac.uk <mailto:d.white@epcc.ed.ac.uk>> wrote: Hi Daniela,
Things seem to be behaving in terms of uploading. Manchester works fine now. How do I register a file as source?
Also, I’m having another issue with my workflow. I was lead to believe that the LSST CVMFS is available to jobs submitted by LSST users? Is that true? I’m having the following error, which causes everything else to fail:
./run-im3shape: line 3: /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/eups/bin/setups.sh: <http://lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/eups/bin/setups.sh:> No such file or directory
Cheers
Darren ------------------------------------------------------------------------- Dr Darren J. White Applications Developer EPCC, University of Edinburgh
Email: d.white@epcc.ed.ac.uk <mailto:d.white@epcc.ed.ac.uk> Telephone: +44 (0) 131 6513415 <tel:0131%20651%203415>
Room 2411, James Clerk Maxwell Building, Kings Buildings, Mayfield Road, Edinburgh EH9 3JZ, United Kingdom -------------------------------------------------------------------------
On 30 Mar 2017, at 16:05, Daniela Bauer <daniela.bauer.grid@googlemail.com <mailto:daniela.bauer.grid@googlemail.com>> wrote:
Hi Darren,
I think the failure + success at Liverpool is DIRAC trying twice and succeeding the second time around. We've been wondering if this is a problem with the source rather than the target SEs. Could you please try and copy the file to UKI-LT2-IC-HEP-disk (QMUL is in downtime) and then try and register it using UKI-LT2-IC-HEP-disk as the source ? It might be better to switch to the command line tools (dirac-dms-replicate-lfn), rather than using the file catalogue cli directly as the command line tools have a debug option (-ddd) which makes it easier to see which bit of the chain fails.
Regards, Daniela
On 30 March 2017 at 13:31, Darren White <d.white@epcc.ed.ac.uk <mailto:d.white@epcc.ed.ac.uk>> wrote: Hi Daniela,
I still have issues making a replica of the uploaded file on Manchester, and I get an error but successful replication at Liverpool:
FC:/lsst/user/d/darren.white>replicate im3shape-grid-eups.tar.gz UKI-NORTHGRID-MAN-HEP-disk __putFile: Failed to put file to storage. globus_ftp_client: the server responded with an error 500 500-Command failed. : System error in name: Permission denied 500-A system call failed: Permission denied 500 End.
__putFile: Failed to put file to storage. globus_ftp_client: the server responded with an error 500 500-Command failed. : System error in name: Permission denied 500-A system call failed: Permission denied 500 End.
Error: Completely failed to replicate file. Failed to replicate with all sources.
FC:/lsst/user/d/darren.white>replicate im3shape-grid-eups.tar.gz UKI-NORTHGRID-LIV-HEP-disk __putFile: Failed to put file to storage. globus_ftp_client: the server responded with an error 500 500-Command failed. : System error in name: Permission denied 500-A system call failed: Permission denied 500 End.
{'Failed': {}, 'Successful': {'/lsst/user/d/darren.white/im3shape-grid-eups.tar.gz': {'register': 0.22239899635314941, 'replicate': 200.95439314842224}}}
File /lsst/user/d/darren.white/im3shape-grid-eups.tar.gz successfully replicated to the UKI-NORTHGRID-LIV-HEP-disk SE
FC:/lsst/user/d/darren.white>replicas im3shape-grid-eups.tar.gz lfn: /lsst/user/d/darren.white/im3shape-grid-eups.tar.gz UKI-LT2-IC-HEP-disk srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//lsst/user/d/darren.white/im3shape-grid-eups.tar.gz <> UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//lsst/user/d/darren.white/im3shape-grid-eups.tar.gz <> UKI-NORTHGRID-LIV-HEP-disk srm://hepgrid11.ph.liv.ac.uk:8446/srm/managerv2?SFN=//lsst/user/d/darren.white/im3shape-grid-eups.tar.gz <>
Cheers
Darren ------------------------------------------------------------------------- Dr Darren J. White Applications Developer EPCC, University of Edinburgh
Email: d.white@epcc.ed.ac.uk <mailto:d.white@epcc.ed.ac.uk> Telephone: +44 (0) 131 6513415 <tel:+44%20131%20651%203415>
Room 2411, James Clerk Maxwell Building, Kings Buildings, Mayfield Road, Edinburgh EH9 3JZ, United Kingdom -------------------------------------------------------------------------
On 30 Mar 2017, at 12:08, Daniela Bauer <daniela.bauer.grid@googlemail.com <mailto:daniela.bauer.grid@googlemail.com>> wrote:
Hi All,
In summary: Manchester and Liverpool were configuration problems. I couldn't reproduce QMUL, so I assume it was a temporary glitch.
Monitoring sites: While we do run basic tests at all sites, testing all sites for all VOs is not within the scope of the DIRAC project and we rely on the VOs to report broken sites. There are efforts to make the ARGO monitoring tool (https://wiki.egi.eu/wiki/ARGO <https://wiki.egi.eu/wiki/ARGO> - yes, I know these webpages aren't very helpful, but it's all I have) available to VOs, so they can automate the monitoring of sites. If you (as a representative of your VO) are interested in this, please talk to your local grid person, or email TB-SUPPORT@jiscmail.ac.uk <mailto:TB-SUPPORT@jiscmail.ac.uk> (there's a number of lists around, this is the is comprehensive one).
Storage space: Liverpool noted, that if LSST wants to use any seizable amount of storage it would need to ask for a space token (i.e. space allocated to LSST, rather than the generic storage pool). Where to stick how much of your data is a complicated issue and way beyond the scope of this list. Again in first order I would talk to your local grid person, who should be able to point you in the right direction (though the right direction might turn out to be tb-support list again).
Darren: Could you please test Liverpool and Manchester and update the GGUS tickets ?
Thanks, Daniela
On 29 March 2017 at 17:44, Darren White <d.white@epcc.ed.ac.uk <mailto:d.white@epcc.ed.ac.uk>> wrote: Hi Daniela,
Fair enough. Glad it seems to be working now. LT2-IC-HEP works too (at least I can replicate the file, which I can’t do for Manchester or Liverpool).
Cheers
Darren ------------------------------------------------------------------------- Dr Darren J. White Applications Developer EPCC, University of Edinburgh
Email: d.white@epcc.ed.ac.uk <mailto:d.white@epcc.ed.ac.uk> Telephone: +44 (0) 131 6513415 <tel:+44%20131%20651%203415>
Room 2411, James Clerk Maxwell Building, Kings Buildings, Mayfield Road, Edinburgh EH9 3JZ, United Kingdom -------------------------------------------------------------------------
On 29 Mar 2017, at 16:21, Daniela Bauer <daniela.bauer.grid@googlemail.com <mailto:daniela.bauer.grid@googlemail.com>> wrote:
Hi Darren,
I think at QMUL was just temporary bad luck with their argus server. The admin had a look and he could not find a reason of what went wrong. I assume UKI-LT2-IC-HEP worked ? As you will have seen I filed tickets for the two sites where I could reproduce the error. But until I know if this is a site problem, a VO problem or a middleware problem I can only speculate. My guess would be some minor form of bit rot as LSST wasn't active for a while. (Both sites seem to work for other VOs.) The only piece of advice I have is to report problems immediately (on this list, or if you are confident using GGUS you can also use GGUS to notify a site directly) and if there's a big gap in usage to expect some hiccups when you start using stuff again. And obviously a retry option for anything you have control over.
Once we have worked out amongst ourselves on what went wrong in Liverpool and Manchester, I will post a summary to the list.
Regards, Daniela
On 29 March 2017 at 15:57, Darren White <d.white@epcc.ed.ac.uk <mailto:d.white@epcc.ed.ac.uk>> wrote: Hi Daniela
That worked this time around! Any idea why it failed? I will soon begin working with partners who are also in the process of joining the LSST VO and wanting to access GridPP. I’ve had quite a few hiccups getting access myself, so It would be useful to know if there are any specific steps we can suggest to them so they can avoid the same pain.
Cheers
Darren ------------------------------------------------------------------------- Dr Darren J. White Applications Developer EPCC, University of Edinburgh
Email: d.white@epcc.ed.ac.uk <mailto:d.white@epcc.ed.ac.uk> Telephone: +44 (0) 131 6513415 <tel:+44%20131%20651%203415>
Room 2411, James Clerk Maxwell Building, Kings Buildings, Mayfield Road, Edinburgh EH9 3JZ, United Kingdom -------------------------------------------------------------------------
On 29 Mar 2017, at 12:14, Daniela Bauer <daniela.bauer.grid@googlemail.com <mailto:daniela.bauer.grid@googlemail.com>> wrote:
Hi Darren,
Can you please also try UKI-LT2-QMUL2-disk again ?
Regards, Daniela
On 28 March 2017 at 16:34, Daniela Bauer <daniela.bauer.grid@googlemail.com <mailto:daniela.bauer.grid@googlemail.com>> wrote: Hi Darren,
can you try and upload a file to UKI-LT2-IC-HEP-disk, please ?
regards, Daniela
On 28 March 2017 at 15:19, Darren White <d.white@epcc.ed.ac.uk <mailto:d.white@epcc.ed.ac.uk>> wrote: Hi there,
I’m trying to upload a file to GridPP using DFC CLI, but I’m getting access errors. I’ve set up my proxy correctly as far as I can tell, but I’m getting errors warning of failure to make directories, or being blacklisted, at the 3 SEs I’ve tried. I had intermittent issues downloading from Manchester in the past, though it seemed to be okay at one point, but now I can’t upload to Manchester, Liverpool or QMUL.
Cheers
Darren
FC:/lsst/user/d/darren.white>add im3shape-grid-eups.tar.gz im3shape-grid-eups.tar.gz UKI-NORTHGRID-MAN-HEP-disk createDirectory: Failed to create directory on storage. srm://bohr3226.tier2.hep.manchester.ac.uk:8446/srm/managerv2?SFN=/dpm/tier2.hep.manchester.ac.uk/home/lsst/lsst/user/d/darren.white: <> SRM2Storage.__gfal_exec(gfal_ls): Execution failed. [SE][Ls][] httpg://bohr3226.tier2.hep.manchester.ac.uk:8446/srm/managerv2: <> CGSI-gSOAP running on localhost reports Error reading token data header: Connection closed
Error: Failed to put file to Storage Element. SRM2Storage.__gfal_exec(gfal_ls): Execution failed. [SE][Ls][] httpg://bohr3226.tier2.hep.manchester.ac.uk:8446/srm/managerv2: <> CGSI-gSOAP running on localhost reports Error reading token data header: Connection closed
FC:/lsst/user/d/darren.white>add im3shape-grid-eups.tar.gz im3shape-grid-eups.tar.gz UKI-NORTHGRID-LIV-HEP-disk __putFile: Failed to put file to storage. file:/tmp/dirac_directory: zero number of replicas
createDirectory: Failed to create directory on storage. srm://hepgrid11.ph.liv.ac.uk:8446/srm/managerv2?SFN=/dpm/ph.liv.ac.uk/home/lsst/lsst/user/d/darren.white: <> file:/tmp/dirac_directory: zero number of replicas
Error: Failed to put file to Storage Element. file:/tmp/dirac_directory: zero number of replicas
FC:/lsst/user/d/darren.white>add im3shape-grid-eups.tar.gz im3shape-grid-eups.tar.gz UKI-LT2-QMUL2-disk __putFile: Failed to put file to storage. [SE][PutDone][SRM_AUTHORIZATION_FAILURE] httpg://se03.esc.qmul.ac.uk:8444/srm/managerv2: <> Request authorization error: user is blacklisted.
createDirectory: Failed to create directory on storage. srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/lsst/lsst/user/d/darren.white: <> [SE][PutDone][SRM_AUTHORIZATION_FAILURE] httpg://se03.esc.qmul.ac.uk:8444/srm/managerv2: <> Request authorization error: user is blacklisted.
Error: Failed to put file to Storage Element. [SE][PutDone][SRM_AUTHORIZATION_FAILURE] httpg://se03.esc.qmul.ac.uk:8444/srm/managerv2: <> Request authorization error: user is blacklisted.
------------------------------------------------------------------------- Dr Darren J. White Applications Developer EPCC, University of Edinburgh
Email: d.white@epcc.ed.ac.uk <mailto:d.white@epcc.ed.ac.uk> Telephone: +44 (0) 131 6513415 <tel:+44%20131%20651%203415>
Room 2411, James Clerk Maxwell Building, Kings Buildings, Mayfield Road, Edinburgh EH9 3JZ, United Kingdom -------------------------------------------------------------------------
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users>
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk <mailto:daniela.bauer@imperial.ac.uk> HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 <tel:+44%2020%207594%207810> http://www.hep.ph.ic.ac.uk/~dbauer/ <http://www.hep.ph.ic.ac.uk/~dbauer/>
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk <mailto:daniela.bauer@imperial.ac.uk> HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 <tel:+44%2020%207594%207810> http://www.hep.ph.ic.ac.uk/~dbauer/ <http://www.hep.ph.ic.ac.uk/~dbauer/>-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users>
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk <mailto:daniela.bauer@imperial.ac.uk> HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 <tel:+44%2020%207594%207810> http://www.hep.ph.ic.ac.uk/~dbauer/ <http://www.hep.ph.ic.ac.uk/~dbauer/>-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users>
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk <mailto:daniela.bauer@imperial.ac.uk> HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 <tel:+44%2020%207594%207810> http://www.hep.ph.ic.ac.uk/~dbauer/ <http://www.hep.ph.ic.ac.uk/~dbauer/>-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users>
-- Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk <mailto:daniela.bauer@imperial.ac.uk> HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 <tel:020%207594%207810> http://www.hep.ph.ic.ac.uk/~dbauer/ <http://www.hep.ph.ic.ac.uk/~dbauer/>-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk <mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users> -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.