Replicas no longer reported correctly by API
******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hello all, It seems like the API does no longer report the replicas of files as it used to. When I do: from DIRAC.Resources.Catalog.FileCatalog import FileCatalog self.fc = FileCatalog() rep = self.fc.getReplicas(lurl) I get: {'OK': True, 'Value': {'Successful': {'/t2k.org/test/t2kdm/test1.txt': {'CA-SFU-T21-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-NORTHGRID-MAN-HEP-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-LT2-QMUL2-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-SOUTHGRID-RALPP-disk': '/t2k.org/test/t2kdm/test1.txt', 'RAL-LCG2-T2K-tape': '/t2k.org/test/t2kdm/test1.txt'}}, 'Failed': {}}} The reply does not contain the actual replicas (i.e. the physical locations), but just the catalogue file names. It used to report the actual physical file paths, and now all our scripts are broken. Was this an intentional change, or is this a bug? If it is intentional, how do I get the physical locations, i.e. the URL to the copies on the storage elements? Cheers Lukas -- Dr. Lukas KOCH Particle Physics Department University of Oxford, United Kingdom Mail: lukas.koch@physics.ox.ac.uk Tel: +44 1865 273449
Hi Lukas, This isn't the intended behaviour: it's the fallback if looking up the PFN fails for any reason (it also doesn't log any extra details when this happens). We'll investigate and get back to you, although it may take a day or two to track it down. Regards, Simon On Wed, Nov 25, 2020 at 12:57:42PM +0000, Lukas Koch wrote:
******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hello all,
It seems like the API does no longer report the replicas of files as it used to.
When I do:
from DIRAC.Resources.Catalog.FileCatalog import FileCatalog self.fc = FileCatalog() rep = self.fc.getReplicas(lurl)
I get:
{'OK': True, 'Value': {'Successful': {'/t2k.org/test/t2kdm/test1.txt': {'CA-SFU-T21-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-NORTHGRID-MAN-HEP-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-LT2-QMUL2-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-SOUTHGRID-RALPP-disk': '/t2k.org/test/t2kdm/test1.txt', 'RAL-LCG2-T2K-tape': '/t2k.org/test/t2kdm/test1.txt'}}, 'Failed': {}}}
The reply does not contain the actual replicas (i.e. the physical locations), but just the catalogue file names. It used to report the actual physical file paths, and now all our scripts are broken.
Was this an intentional change, or is this a bug? If it is intentional, how do I get the physical locations, i.e. the URL to the copies on the storage elements?
Cheers Lukas
Hi Lukas, The getReplicas function on the FileCatalog object only returned the PFN if it was stored in the database: This behaviour has been deprecated for a very long time (because it only returned the protocol that the file was written with, even on SEs that had other protocols available). The PFN column is now not requested by default as they continue to work towards removing it entirely. The outer bit of the code puts the LFN in instead as a placeholder. Are you able to use the public API getReplicas function? It's in the Dirac object: from DIRAC.Interfaces.API.Dirac import Dirac dirac = Dirac() print dirac.getReplicas('/t2k.org/test/t2kdm/test1.txt') The functions in that package are considered to be the stable interface by the upstream developers, so shouldn't change without a very good reason and lots of warning. Regards, Simon On Wed, Nov 25, 2020 at 02:11:33PM +0000, Simon Fayer wrote:
Hi Lukas,
This isn't the intended behaviour: it's the fallback if looking up the PFN fails for any reason (it also doesn't log any extra details when this happens).
We'll investigate and get back to you, although it may take a day or two to track it down.
Regards, Simon
On Wed, Nov 25, 2020 at 12:57:42PM +0000, Lukas Koch wrote:
******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hello all,
It seems like the API does no longer report the replicas of files as it used to.
When I do:
from DIRAC.Resources.Catalog.FileCatalog import FileCatalog self.fc = FileCatalog() rep = self.fc.getReplicas(lurl)
I get:
{'OK': True, 'Value': {'Successful': {'/t2k.org/test/t2kdm/test1.txt': {'CA-SFU-T21-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-NORTHGRID-MAN-HEP-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-LT2-QMUL2-disk': '/t2k.org/test/t2kdm/test1.txt', 'UKI-SOUTHGRID-RALPP-disk': '/t2k.org/test/t2kdm/test1.txt', 'RAL-LCG2-T2K-tape': '/t2k.org/test/t2kdm/test1.txt'}}, 'Failed': {}}}
The reply does not contain the actual replicas (i.e. the physical locations), but just the catalogue file names. It used to report the actual physical file paths, and now all our scripts are broken.
Was this an intentional change, or is this a bug? If it is intentional, how do I get the physical locations, i.e. the URL to the copies on the storage elements?
Cheers Lukas
participants (2)
-
Lukas Koch
-
Simon Fayer