- Gridpp-Dirac-Users - imperial.ac.uk

Questions on dirac file catolog
by Groves, Andrew 18 Dec '20

18 Dec '20

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hi again Im trying to remove files on mass from the dirac file catolog (a whole directories worth) i've tried the command dirac-dms-clean-directory however I keep getting the error Failed to perform isDirectory from any catolog. Im trying dirac-dms-clean-directory /mu3e.org/prod/build4.4.1. Also im trying to use dirac-dms-data-size, could I do this on the whole directory in one command or only on a list of files. Thanks Andrew Putting hear as may help others too

3 4

Problem instrall Dirac UI
by Groves, Andrew 16 Dec '20

16 Dec '20

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hi Im trying to install the dirac_ui with the line wget -np -O dirac-install https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/Core/scripts/… and it getting a error that it isnt found? Is there a new one I am missing? Thanks Andrew

2 2

GridPP DIRAC UI on cvmfs
by Daniela Bauer 03 Dec '20

03 Dec '20

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hi All, As we are aware that setting up a DIRAC UI is not always straightforward for our users we are currently trying to deploy a GridPP specific DIRAC UI on cvmfs. To access this UI your machine needs to have access to /cvmfs/dirac.egi.eu *and* /cvmfs/grid.cern.ch If you have access to both these repositories you can try it out by sourcing (in a clean terminal!): source /cvmfs/dirac.egi.eu/dirac/bashrc_gridpp Please let me know of any issues you find. Regards, Daniela -- Sent from my guinea pig enhanced living room ----------------------------------------------------------- daniela.bauer(a)imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: Working from home, please use email. http://www.hep.ph.ic.ac.uk/~dbauer/

1 0

Strange warnings
by Groves, Andrew 03 Dec '20

03 Dec '20

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hi I am getting some strange warnings. The first one is when I am linking to my proxy dirac-proxy-init -g mu3e.org_user -M Generating proxy... Enter Certificate password: Your CRLs appear to be outdated, but you have no access to update them. Added VOMS attribute /mu3e.org and after running a load of jobs Can't verify proxy or certificate file /user/agroves/.globus/x509up_u10233:No chain loaded ( 1106 : ) Job submission failure Can't verify file /user/agroves/.globus/x509up_u10233:No chain loaded ( 1106 : ) Traceback (most recent call last): File "Mu3eGRIDSubmission.py", line 93, in <module> setup_submit(i,frames,local,GroupName,mode,rate) File "Mu3eGRIDSubmission.py", line 61, in setup_submit jobid = result['JobID'] KeyError: 'JobID' Any help? Thanks Andrew

2 1

Re: [Gridpp-Dirac-Users] Fwd: Re: Replicas no longer reported correctly by API
by Lukas Koch 25 Nov '20

25 Nov '20

Hello Simon, I am not on the mailing list, so I'm afraid you need to use "reply all" to reach me as well. Sorry for the inconvenience. The solution you suggested works. Thanks! Is this "stable" API documented somewhere? It has been a while since I last did any developing on our grid scripts, but it was a bit hard to figure out how to do things with the API. At least back then I could not find any docs explaining how to do these things. Cheers Lukas On 25/11/2020 16:51, Sophie King wrote: > > > > -------- Forwarded Message -------- > Subject: Re: [Gridpp-Dirac-Users] Replicas no longer reported correctly > by API > Date: Wed, 25 Nov 2020 14:44:06 +0000 > From: Simon Fayer <simon.fayer05(a)imperial.ac.uk> > Reply-To: gridpp-dirac-users(a)imperial.ac.uk > To: gridpp-dirac-users(a)imperial.ac.uk > > > > Hi Lukas, > > The getReplicas function on the FileCatalog object only returned the PFN if > it was stored in the database: This behaviour has been deprecated for a > very long time (because it only returned the protocol that the file was > written with, even on SEs that had other protocols available). > > The PFN column is now not requested by default as they continue to work > towards removing it entirely. The outer bit of the code puts the LFN in > instead as a placeholder. > > Are you able to use the public API getReplicas function? It's in the > Dirac object: > > from DIRAC.Interfaces.API.Dirac import Dirac > dirac = Dirac() > print dirac.getReplicas('/t2k.org/test/t2kdm/test1.txt') > > The functions in that package are considered to be the stable interface by > the upstream developers, so shouldn't change without a very good reason and > lots of warning. > > Regards, > Simon > > > On Wed, Nov 25, 2020 at 02:11:33PM +0000, Simon Fayer wrote: >> Hi Lukas, >> >> This isn't the intended behaviour: it's the fallback if looking up the PFN >> fails for any reason (it also doesn't log any extra details when this >> happens). >> >> We'll investigate and get back to you, although it may take a day or >> two to >> track it down. >> >> Regards, >> Simon >> >> >> On Wed, Nov 25, 2020 at 12:57:42PM +0000, Lukas Koch wrote: >> > > ******************* >> > This email originates from outside Imperial. Do not click on links and >> > attachments unless you recognise the sender. If you trust the >> sender, add >> > them to your safe senders list >> > https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email >> stamping for >> > this address. >> > ******************* >> > Hello all, >> > > It seems like the API does no longer report the replicas of files >> as it used >> > to. >> > > When I do: >> > > from DIRAC.Resources.Catalog.FileCatalog import FileCatalog >> > self.fc = FileCatalog() >> > rep = self.fc.getReplicas(lurl) >> > > I get: >> > > {'OK': True, 'Value': {'Successful': {'/t2k.org/test/t2kdm/test1.txt': >> > {'CA-SFU-T21-disk': '/t2k.org/test/t2kdm/test1.txt', >> > 'UKI-NORTHGRID-MAN-HEP-disk': '/t2k.org/test/t2kdm/test1.txt', >> > 'UKI-LT2-QMUL2-disk': '/t2k.org/test/t2kdm/test1.txt', >> > 'UKI-SOUTHGRID-RALPP-disk': '/t2k.org/test/t2kdm/test1.txt', >> > 'RAL-LCG2-T2K-tape': '/t2k.org/test/t2kdm/test1.txt'}}, 'Failed': {}}} >> > > The reply does not contain the actual replicas (i.e. the physical >> > locations), but just the catalogue file names. It used to report the >> actual >> > physical file paths, and now all our scripts are broken. >> > > Was this an intentional change, or is this a bug? If it is >> intentional, how >> > do I get the physical locations, i.e. the URL to the copies on the >> storage >> > elements? >> > > Cheers >> > Lukas >> > > > -- > _______________________________________________ > Gridpp-Dirac-Users mailing list > Gridpp-Dirac-Users(a)imperial.ac.uk > https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users > -- Dr. Lukas KOCH Particle Physics Department University of Oxford, United Kingdom Mail: lukas.koch(a)physics.ox.ac.uk Tel: +44 1865 273449

2 1

Replicas no longer reported correctly by API
by Lukas Koch 25 Nov '20

25 Nov '20

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hello all, It seems like the API does no longer report the replicas of files as it used to. When I do: from DIRAC.Resources.Catalog.FileCatalog import FileCatalog self.fc = FileCatalog() rep = self.fc.getReplicas(lurl) I get: {'OK': True, 'Value': {'Successful': {'/t2k.org/test/t2kdm/test1.txt': {'CA-SFU-T21-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-NORTHGRID-MAN-HEP-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-LT2-QMUL2-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-SOUTHGRID-RALPP-disk': '/t2k.org/test/t2kdm/test1.txt', 'RAL-LCG2-T2K-tape': '/t2k.org/test/t2kdm/test1.txt'}}, 'Failed': {}}} The reply does not contain the actual replicas (i.e. the physical locations), but just the catalogue file names. It used to report the actual physical file paths, and now all our scripts are broken. Was this an intentional change, or is this a bug? If it is intentional, how do I get the physical locations, i.e. the URL to the copies on the storage elements? Cheers Lukas -- Dr. Lukas KOCH Particle Physics Department University of Oxford, United Kingdom Mail: lukas.koch(a)physics.ox.ac.uk Tel: +44 1865 273449

2 2

DIRAC upgrade complete
by Daniela Bauer 18 Nov '20

18 Nov '20

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Dear All, We have finished the DIRAC upgrade and the server should be usable again. This was a major upgrade and you will have to upgrade your UIs (apologies). Please find the instructions/version in the usual place: https://www.gridpp.ac.uk/wiki/Quick_Guide_to_Dirac#Dirac_client_installation If you find any issues, please email this list as it is likely other people might see it as well. For LZ only: Submission to the cloud currently only works when using JDLs (not the API). We are working on it. Regards, Daniela -- Sent from my guinea pig enhanced living room ----------------------------------------------------------- daniela.bauer(a)imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: Working from home, please use email. http://www.hep.ph.ic.ac.uk/~dbauer/

4 7

The DIRAC upgrade is happening *now*
by Daniela Bauer 16 Nov '20

16 Nov '20

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Dear All, we'll let you now when we are done. Daniela -- Sent from my guinea pig enhanced living room ----------------------------------------------------------- daniela.bauer(a)imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: Working from home, please use email. http://www.hep.ph.ic.ac.uk/~dbauer/

1 0

Advance Warning: Major version change in GridPP DIRAC
by Daniela Bauer 10 Nov '20

10 Nov '20

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Dear All, The GridPP DIRAC instance is due for a major upgrade, as the old version is going out of support. The upgrade is planned for Monday 16. November 2020. What does this mean for you ? - The upgrade is not backwards compatible. You will need to update your DIRAC UI. We will advise you of which version to use a couple of days before the upgrade. You cannot upgrade beforehand as the new version will not be compatible with our current setup. - Any jobs that have not finished processing on upgrade day will be lost. We recommend stopping submission a couple of days in advance. - There should be little difference in how to interact with GridPP DIRAC afterwards, but you will have to re-upload your proxy after the upgrade as the way DIRAC handles proxies will change. - The data catalogue will be unaffected. And no, we as the maintainers are not looking forward to this either. Regards, Daniela -- Sent from my guinea pig enhanced living room ----------------------------------------------------------- daniela.bauer(a)imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: Working from home, please use email. http://www.hep.ph.ic.ac.uk/~dbauer/

1 1

Re: [Gridpp-Dirac-Users] FW: Dirac register file in file catalogue
by Daniela Bauer 07 Oct '20

07 Oct '20

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hi Priyaa, just to clarify, when working with DIRAC we do not recommend using gfal-copy and the DIRAC register mechanism separately. We only recommend registering data explicitly (as it tends to be error prone) if they are already on a storage element (or if there is no way to upload them with DIRAC). dirac-dms-put-and-register-request is not quite what it seems. The help throws up an unfortunately not very clear warning: warning: make sure the file you want to put is accessible from DIRAC production hosts, i.e. put file on network fs (AFS or NFS), otherwise operation will fail!!! This looks like some CERN centric hangover to me. This would only work if the file is on the actual DIRAC server. Unfortunately it will probably hang around forever now. The command you really want is dirac-dms-add-file - it will upload the file and register it in the DIRAC catalogue all in one go. Regards, Daniela On Tue, 6 Oct 2020 at 15:20, Priyaa Thavasimani < priyaa.thavasimani(a)manchester.ac.uk> wrote: > Hi Daniela, > > I have a question regarding registering file without using > dirac-dms-add-file. As you suggested, I have gfal-cop(y) ied a local file > to pfn and registered using “register” command. It is working fine. But I > also thought is there a dirac command that combines the functionality of > gfal-copy and register? Would like to know the difference between > dirac-dms-put-and-register-request and dirac-dms-add-file? I got to know > about dirac-dms-put-register-request, and submitted the request, but keeps > on waiting even for a small file(assuming it does the functionality, also > there is no clear documentation about this as far as I am aware). It would > be great if you could clarify this too. > > > > Here is the request status for dirac-dms-put-and-register-request: > > > > dirac-rms-request req1 > > Request name='req1' ID=319813 Status='Waiting' > > Created 2020-10-06 11:48:41, Updated 2020-10-06 11:48:41, NotBefore > 2020-10-06 11:48:41 > > Owner: '/C=UK/O=eScience/OU=Manchester/L=HEP/CN=priyaa thavasimani', > Group: skatelescope.eu_user > > [0] Operation Type='PutAndRegister' ID=290706 Order=0 Status='Waiting' > > TargetSE: UKI-NORTHGRID-MAN-HEP-disk - Created 2020-10-06 11:48:41, > Updated 2020-10-06 11:48:41 > > [01] ID=17727 LFN='/ > skatelescope.eu/user/p/priyaa.thavasimani/ramdom.txt' Status='Waiting' > Checksum='01b700bc' > > > > Thanks, > > Priyaa > > > > *From: *Priyaa Thavasimani <priyaa.thavasimani(a)manchester.ac.uk> > *Date: *Wednesday, 1 July 2020 at 12:53 > *To: *Daniela Bauer <daniela.bauer(a)IMPERIAL.AC.UK> > *Subject: *Dirac register file in file catalogue > > > > Hi Daniela, > > I am a Research Associate working in JBCA, University of Manchester. > We have 1.3 TB data in our highmem machine to be registered in the file > catalogue. Have seen few of the links below to do that. > > 1. > https://dirac.readthedocs.io/en/latest/UserGuide/CommandReference/DataManag… > 2. https://groups.google.com/forum/#!topic/diracgrid-forum/sclcLrQBPFY > > But the links are not so much helpful. It would be great if you could > guide on this. I am wondering if it is possible register the file in file > catalog without adding the file in LFN using "dirac-dms-add-file"? If so, > could you please send me some links that could help me out to do the same. > > > > Thanks, > > Priyaa > -- Sent from my guinea pig enhanced living room ----------------------------------------------------------- daniela.bauer(a)imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: Working from home, please use email. http://www.hep.ph.ic.ac.uk/~dbauer/

1 0