Hi Daniela,

I’ve managed to get the lsst.opensciencegrid CVMFS mounted on CernVM for now, and I was able to make the changes needed to my workflow. I was able to get a few jobs to run (and failing for unrelated reasons), so thanks for all your help! I will email Alessandra directly about other things to save everyone on this list. I do have a generic question though, which you may be able to answer, that I’ve not been able to find out directly. How do you know which CVMFS repos are mounted for which VOs, and at which sites? It seems like something that is important for a person to know that wants to run things against CVMFS, but I can’t find the info anywhere.

Cheers

Darren

-------------------------------------------------------------------------
Dr Darren J. White 
Applications Developer 
EPCC, University of Edinburgh 

Email: d.white@epcc.ed.ac.uk
Telephone: +44 (0) 131 6513415

Room 2411, James Clerk Maxwell Building,
Kings Buildings, Mayfield Road,
Edinburgh EH9 3JZ, United Kingdom
-------------------------------------------------------------------------

On 4 Apr 2017, at 12:23, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:

Hi Darren,

I think Alessandra (from Manchester) is now on this list. Maybe she has the scripts that the LSST jobs used to use to run on GridPP. I can't comment on CERNVM from here, sorry.

Regards,
Daniela

On 4 April 2017 at 12:15, Darren White <d.white@epcc.ed.ac.uk> wrote:
Hi Sam, Daniela

Thanks for the response. 

"Does that file exist in other instances of the LSST CVMFS repo?” - I’m not sure what you mean by that?

"Is the CVMFS repo for LSST available at the site the job is running at?” - I have no idea how to check. I’m still a bit of a newbie when it comes to GridPP. I was given a workflow that used to work, but now doesn’t, and I’ve been using it to help understand how GridPP works. Given the issues, it actually has, but it’s not been too clear at times how to do certain things. I was told by our local GridPP person that the sites that accept jobs from LSST VOs should have the LSST CVMFS repo available. The only reference I could find that details how to connect to an LSST repo is at https://github.com/airnandez/lsst-cvmfs, which is what I used to test the workflow locally before uploading to the grid (which is where this thread started, when uploading the new workflow that did work with the lsst.in2ps.fr CVMFS repo).

I can’t find any info on how to connect my CernVM to the lsst.opensciencegrid.org in order to know what path to source to enable the workflow to use EUPS. Any information would be greatly appreciated.

Cheers

Darren

-------------------------------------------------------------------------
Dr Darren J. White 
Applications Developer 
EPCC, University of Edinburgh 

Email: d.white@epcc.ed.ac.uk
Telephone: +44 (0) 131 6513415

Room 2411, James Clerk Maxwell Building,
Kings Buildings, Mayfield Road,
Edinburgh EH9 3JZ, United Kingdom
-------------------------------------------------------------------------

On 4 Apr 2017, at 12:04, Sam Skipsey <sskipsey@googlemail.com> wrote:

Strictly, the LSST CVMFS repo will be available for jobs submitted as any user (it's not a protected filesystem), as long as the repo is enabled at the site. Does that file exist in other instances of the LSST CVMFS repo? Is the CVMFS repo for LSST available at the site the job is running at?

Sam


On Tue, 4 Apr 2017 at 11:56 Darren White <d.white@epcc.ed.ac.uk> wrote:
Hi Daniela,

Things seem to be behaving in terms of uploading. Manchester works fine now. How do I register a file as source?

Also, I’m having another issue with my workflow. I was lead to believe that the LSST CVMFS is available to jobs submitted by LSST users? Is that true? I’m having the following error, which causes everything else to fail:

./run-im3shape: line 3: /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/eups/bin/setups.sh: No such file or directory

Cheers

Darren
-------------------------------------------------------------------------
Dr Darren J. White 
Applications Developer 
EPCC, University of Edinburgh 

Email: d.white@epcc.ed.ac.uk
Telephone: +44 (0) 131 6513415

Room 2411, James Clerk Maxwell Building,
Kings Buildings, Mayfield Road,
Edinburgh EH9 3JZ, United Kingdom
-------------------------------------------------------------------------

On 30 Mar 2017, at 16:05, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:

Hi Darren,

I think the failure + success at Liverpool is DIRAC trying twice and succeeding the second time around.
We've been wondering if this is a problem with the source rather than the target SEs. Could you please try and copy the file to UKI-LT2-IC-HEP-disk (QMUL is in downtime) and then try and register it using UKI-LT2-IC-HEP-disk as the source ?
It might be better to switch to the command line tools (dirac-dms-replicate-lfn), rather than using the file catalogue cli directly as the command line tools have a debug option (-ddd) which makes it easier to see which bit of the chain fails.

Regards,
Daniela


On 30 March 2017 at 13:31, Darren White <d.white@epcc.ed.ac.uk> wrote:
Hi Daniela,

I still have issues making a replica of the uploaded file on Manchester, and I get an error but successful replication at Liverpool:

FC:/lsst/user/d/darren.white>replicate im3shape-grid-eups.tar.gz UKI-NORTHGRID-MAN-HEP-disk
__putFile: Failed to put file to storage. globus_ftp_client: the server responded with an error
500 500-Command failed. : System error in name: Permission denied
500-A system call failed: Permission denied
500 End.

__putFile: Failed to put file to storage. globus_ftp_client: the server responded with an error
500 500-Command failed. : System error in name: Permission denied
500-A system call failed: Permission denied
500 End.

Error: Completely failed to replicate file. Failed to replicate with all sources.

FC:/lsst/user/d/darren.white>replicate im3shape-grid-eups.tar.gz UKI-NORTHGRID-LIV-HEP-disk
__putFile: Failed to put file to storage. globus_ftp_client: the server responded with an error
500 500-Command failed. : System error in name: Permission denied
500-A system call failed: Permission denied
500 End.

{'Failed': {},
 'Successful': {'/lsst/user/d/darren.white/im3shape-grid-eups.tar.gz': {'register': 0.22239899635314941,
                                                                        'replicate': 200.95439314842224}}}

File /lsst/user/d/darren.white/im3shape-grid-eups.tar.gz successfully replicated to the UKI-NORTHGRID-LIV-HEP-disk SE

FC:/lsst/user/d/darren.white>replicas im3shape-grid-eups.tar.gz 
lfn: /lsst/user/d/darren.white/im3shape-grid-eups.tar.gz
UKI-LT2-IC-HEP-disk srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=//lsst/user/d/darren.white/im3shape-grid-eups.tar.gz
UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=//lsst/user/d/darren.white/im3shape-grid-eups.tar.gz
UKI-NORTHGRID-LIV-HEP-disk srm://hepgrid11.ph.liv.ac.uk:8446/srm/managerv2?SFN=//lsst/user/d/darren.white/im3shape-grid-eups.tar.gz

Cheers

Darren
-------------------------------------------------------------------------
Dr Darren J. White 
Applications Developer 
EPCC, University of Edinburgh 

Email: d.white@epcc.ed.ac.uk
Telephone: +44 (0) 131 6513415

Room 2411, James Clerk Maxwell Building,
Kings Buildings, Mayfield Road,
Edinburgh EH9 3JZ, United Kingdom
-------------------------------------------------------------------------

On 30 Mar 2017, at 12:08, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:

Hi All,

In summary:
Manchester and Liverpool were configuration problems. I couldn't reproduce QMUL, so I assume it was a temporary glitch.

Monitoring sites:
While we do run basic tests at all sites, testing all sites for all VOs is not within the scope of the DIRAC project and we rely on the VOs to report broken sites. There are efforts to make the ARGO monitoring tool (https://wiki.egi.eu/wiki/ARGO - yes, I know these webpages aren't very helpful, but it's all I have) available to VOs, so they can automate the monitoring of sites. If you (as a representative of your VO) are interested in this, please talk to your local grid person, or email TB-SUPPORT@jiscmail.ac.uk (there's a number of lists around, this is the is comprehensive one).

Storage space:
Liverpool noted, that if LSST wants to use any seizable amount of storage it would need to ask for a space token (i.e. space allocated to LSST, rather than the generic storage pool). Where to stick how much of your data is a complicated issue and way beyond the scope of this list. Again in first order I would talk to your local grid person, who should be able to point you in the right direction (though the right direction might turn out to be tb-support list again).
 
Darren: Could you please test Liverpool and Manchester and update the GGUS tickets ?

Thanks,
Daniela

On 29 March 2017 at 17:44, Darren White <d.white@epcc.ed.ac.uk> wrote:
Hi Daniela,

Fair enough. Glad it seems to be working now. LT2-IC-HEP works too (at least I can replicate the file, which I can’t do for Manchester or Liverpool).

Cheers

Darren
-------------------------------------------------------------------------
Dr Darren J. White 
Applications Developer 
EPCC, University of Edinburgh 

Email: d.white@epcc.ed.ac.uk
Telephone: +44 (0) 131 6513415

Room 2411, James Clerk Maxwell Building,
Kings Buildings, Mayfield Road,
Edinburgh EH9 3JZ, United Kingdom
-------------------------------------------------------------------------

On 29 Mar 2017, at 16:21, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:

Hi Darren,

I think at QMUL was just temporary bad luck with their argus server. The admin had a look and he could not find a reason of what went wrong. I assume UKI-LT2-IC-HEP worked ? As you will have seen I filed tickets for the two sites where I could reproduce the error. But until I know if this is a site problem, a VO problem or a middleware problem I can only speculate. My guess would be some minor form of bit rot as LSST wasn't active for a while. (Both sites seem to work for other VOs.)
The only piece of advice I have is to report problems immediately (on this list, or if you are confident using GGUS you can also use GGUS to notify a site directly) and if there's a big gap in usage to expect some hiccups when you start using stuff again.
And obviously a retry option for anything you have control over.

Once we have worked out amongst ourselves on what went wrong in Liverpool and Manchester, I will post a summary to the list.

Regards,
Daniela

On 29 March 2017 at 15:57, Darren White <d.white@epcc.ed.ac.uk> wrote:
Hi Daniela

That worked this time around! Any idea why it failed? I will soon begin working with partners who are also in the process of joining the LSST VO and wanting to access GridPP. I’ve had quite a few hiccups getting access myself, so It would be useful to know if there are any specific steps we can suggest to them so they can avoid the same pain.

Cheers

Darren
-------------------------------------------------------------------------
Dr Darren J. White 
Applications Developer 
EPCC, University of Edinburgh 

Email: d.white@epcc.ed.ac.uk
Telephone: +44 (0) 131 6513415

Room 2411, James Clerk Maxwell Building,
Kings Buildings, Mayfield Road,
Edinburgh EH9 3JZ, United Kingdom
-------------------------------------------------------------------------

On 29 Mar 2017, at 12:14, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:

Hi Darren,

Can you please also try UKI-LT2-QMUL2-disk again ? 

Regards,
Daniela

On 28 March 2017 at 16:34, Daniela Bauer <daniela.bauer.grid@googlemail.com> wrote:
Hi Darren,

can you try and upload a file to UKI-LT2-IC-HEP-disk, please ?

regards,
Daniela

On 28 March 2017 at 15:19, Darren White <d.white@epcc.ed.ac.uk> wrote:
Hi there,

I’m trying to upload a file to GridPP using DFC CLI, but I’m getting access errors. I’ve set up my proxy correctly as far as I can tell, but I’m getting errors warning of failure to make directories, or being blacklisted, at the 3 SEs I’ve tried. I had intermittent issues downloading from Manchester in the past, though it seemed to be okay at one point, but now I can’t upload to Manchester, Liverpool or QMUL. 

Cheers

Darren

FC:/lsst/user/d/darren.white>add im3shape-grid-eups.tar.gz im3shape-grid-eups.tar.gz UKI-NORTHGRID-MAN-HEP-disk
createDirectory: Failed to create directory on storage. 
srm://bohr3226.tier2.hep.manchester.ac.uk:8446/srm/managerv2?SFN=/dpm/tier2.hep.manchester.ac.uk/home/lsst/lsst/user/d/darren.white: 
SRM2Storage.__gfal_exec(gfal_ls): Execution failed. [SE][Ls][] httpg://bohr3226.tier2.hep.manchester.ac.uk:8446/srm/managerv2: CGSI-gSOAP running on localhost reports Error reading token data header: Connection closed

Error: Failed to put file to Storage Element. SRM2Storage.__gfal_exec(gfal_ls): Execution failed. [SE][Ls][] httpg://bohr3226.tier2.hep.manchester.ac.uk:8446/srm/managerv2: CGSI-gSOAP running on localhost reports Error reading token data header: Connection closed

FC:/lsst/user/d/darren.white>add im3shape-grid-eups.tar.gz im3shape-grid-eups.tar.gz UKI-NORTHGRID-LIV-HEP-disk
__putFile: Failed to put file to storage. file:/tmp/dirac_directory: zero number of replicas

createDirectory: Failed to create directory on storage. 
srm://hepgrid11.ph.liv.ac.uk:8446/srm/managerv2?SFN=/dpm/ph.liv.ac.uk/home/lsst/lsst/user/d/darren.white: 
file:/tmp/dirac_directory: zero number of replicas

Error: Failed to put file to Storage Element. file:/tmp/dirac_directory: zero number of replicas

FC:/lsst/user/d/darren.white>add im3shape-grid-eups.tar.gz im3shape-grid-eups.tar.gz UKI-LT2-QMUL2-disk        
__putFile: Failed to put file to storage. [SE][PutDone][SRM_AUTHORIZATION_FAILURE] httpg://se03.esc.qmul.ac.uk:8444/srm/managerv2: Request authorization error: user is blacklisted.

createDirectory: Failed to create directory on storage. 
srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/lsst/lsst/user/d/darren.white: 
[SE][PutDone][SRM_AUTHORIZATION_FAILURE] httpg://se03.esc.qmul.ac.uk:8444/srm/managerv2: Request authorization error: user is blacklisted.

Error: Failed to put file to Storage Element. [SE][PutDone][SRM_AUTHORIZATION_FAILURE] httpg://se03.esc.qmul.ac.uk:8444/srm/managerv2: Request authorization error: user is blacklisted.

-------------------------------------------------------------------------
Dr Darren J. White 
Applications Developer 
EPCC, University of Edinburgh 

Email: d.white@epcc.ed.ac.uk
Telephone: +44 (0) 131 6513415

Room 2411, James Clerk Maxwell Building,
Kings Buildings, Mayfield Road,
Edinburgh EH9 3JZ, United Kingdom
-------------------------------------------------------------------------


The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users




--
Sent from the pit of despair

-----------------------------------------------------------
daniela.bauer@imperial.ac.uk
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: +44-(0)20-75947810
http://www.hep.ph.ic.ac.uk/~dbauer/



--
Sent from the pit of despair

-----------------------------------------------------------
daniela.bauer@imperial.ac.uk
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: +44-(0)20-75947810
http://www.hep.ph.ic.ac.uk/~dbauer/
--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users


The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users




--
Sent from the pit of despair

-----------------------------------------------------------
daniela.bauer@imperial.ac.uk
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: +44-(0)20-75947810
http://www.hep.ph.ic.ac.uk/~dbauer/
--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users


The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users




--
Sent from the pit of despair

-----------------------------------------------------------
daniela.bauer@imperial.ac.uk
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: +44-(0)20-75947810
http://www.hep.ph.ic.ac.uk/~dbauer/
--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users


The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users




--
Sent from the pit of despair

-----------------------------------------------------------
daniela.bauer@imperial.ac.uk
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: +44-(0)20-75947810
http://www.hep.ph.ic.ac.uk/~dbauer/
--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users


The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users




--
Sent from the pit of despair

-----------------------------------------------------------
daniela.bauer@imperial.ac.uk
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: +44-(0)20-75947810
http://www.hep.ph.ic.ac.uk/~dbauer/
--
_______________________________________________
Gridpp-Dirac-Users mailing list
Gridpp-Dirac-Users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users