******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hi again Im trying to remove files on mass from the dirac file catolog (a whole directories worth) i've tried the command dirac-dms-clean-directory however I keep getting the error Failed to perform isDirectory from any catolog. Im trying dirac-dms-clean-directory /mu3e.org/prod/build4.4.1. Also im trying to use dirac-dms-data-size, could I do this on the whole directory in one command or only on a list of files. Thanks Andrew Putting hear as may help others too
Hi Andrew, On Thu, Dec 17, 2020 at 04:46:49PM +0000, Groves, Andrew wrote:
Im trying to remove files on mass from the dirac file catolog (a whole directories worth) i've tried the command dirac-dms-clean-directory however I keep getting the error Failed to perform isDirectory from any catolog.
Im trying dirac-dms-clean-directory /mu3e.org/prod/build4.4.1.
Hmm, I think this should work (I tried it out on a small test case and verified the permissions on the directory you're trying to remove). Can you please verify that your proxy is valid and in the correct group? If the proxy looks OK, try adding the -ddd option and see if the extra debug gives you any further clues as to what the problem may be.
Also im trying to use dirac-dms-data-size, could I do this on the whole directory in one command or only on a list of files.
It can only take a list of files, but you can give it a file containing a list of files (which you can generate with another command): dirac-dms-find-lfns Path=/mu3e.org/... > files.lst dirac-dms-data-size files.lst Regards, Simon
******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hi Thanks for than. My proxy should be correct (I have used the same command to delete the another directory) however for this quite sizeable one is still throws a error. Ive included the debug for it as I don't understand it. Is there a limit on the size which it will work? Thanks Andrew [groves@gamma.ph.liv.ac.uk GRID]> dirac-dms-clean-directory /mu3e.org/prod/build4.4.1/BhabbaMichel -ddd 2020-12-18 10:31:56 UTC Framework [139810673047360] DEBUG: dirac.cfg should be at /user/agroves/dirac_ui/etc/dirac.cfg 2020-12-18 10:31:56 UTC Framework [139810673047360] DEBUG: CFG merged 2020-12-18 10:31:56 UTC Framework [139810673047360] DEBUG: Updating configuration internals 2020-12-18 10:31:56 UTC Framework [139810673047360] DEBUG: Updating configuration internals Parsing command line Trying to load Resources.LogBackends.StdoutBackend Trying to load DIRAC.Resources.LogBackends.StdoutBackend 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Refreshing configuration... 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Refreshing from list ['dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server', 'dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server', 'dips://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server'] 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Randomized server list is dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server, dips://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server, dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Already given a valid url dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to refresh from dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to connect to: dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Using proxy /user/agroves/.globus/x509up_u10233 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: CAs location is /cvmfs/grid.cern.ch/etc/grid-security/certificates 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Loaded 94 CAs [188 CRLs] 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Connection timeout set to: 1 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Authenticated peer (/C=UK/O=eScience/OU=Imperial/L=Physics/CN=dirac02.grid.hep.ph.ic.ac.uk) 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: New session connecting to server at ('dirac02.grid.hep.ph.ic.ac.uk', 9135) 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Connected to: dips://dirac02.grid.hep.ph.ic.ac.uk:9135/Configuration/Server 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: New connection -> 2a0c:5bc0:c8:2:216:3eff:fe7f:1c0:9135 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Closing socket 2020-12-18 10:31:56 UTC dirac-dms-clean-directory [139810673047360] DEBUG: New version available Updating to version 2020-12-18 09:33:23.654869... 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Updating configuration internals 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Updated to version 2020-12-18 09:33:23.654869 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to load Resources.LogBackends.StdoutBackend 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to load WebAppDIRAC.Resources.LogBackends.StdoutBackend 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to load GridPPDIRAC.Resources.LogBackends.StdoutBackend 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to load VMDIRAC.Resources.LogBackends.StdoutBackend 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to load DIRAC.Resources.LogBackends.StdoutBackend 2020-12-18 10:31:57 UTC dirac-dms-clean-directory/FileCatalogFactory [139810673047360] DEBUG: Creating FileCatalog client of type FileCatalog 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to load Resources.Catalog.FileCatalogClient 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to load WebAppDIRAC.Resources.Catalog.FileCatalogClient 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to load GridPPDIRAC.Resources.Catalog.FileCatalogClient 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to load VMDIRAC.Resources.Catalog.FileCatalogClient 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to load DIRAC.Resources.Catalog.FileCatalogClient 2020-12-18 10:31:57 UTC dirac-dms-clean-directory/FileCatalogFactory [139810673047360] DEBUG: Loaded module FileCatalogClient 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] NOTICE: Cleaning directory '/mu3e.org/prod/build4.4.1/BhabbaMichel' ... 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Discovering URL for service DataManagement/FileCatalog -> dips://dirac01.grid.hep.ph.ic.ac.uk:9197/DataManagement/FileCatalog 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to connect to: dips://dirac01.grid.hep.ph.ic.ac.uk:9197/DataManagement/FileCatalog 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Using proxy /user/agroves/.globus/x509up_u10233 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Connection timeout set to: 1 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Authenticated peer (/C=UK/O=eScience/OU=Imperial/L=Physics/CN=dirac01.grid.hep.ph.ic.ac.uk) 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: New session connecting to server at ('dirac01.grid.hep.ph.ic.ac.uk', 9197) 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Connected to: dips://dirac01.grid.hep.ph.ic.ac.uk:9197/DataManagement/FileCatalog 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: New connection -> 2a0c:5bc0:c8:2:216:3eff:fe7f:1bf:9197 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Closing socket 2020-12-18 10:31:57 UTC dirac-dms-clean-directory/DataManager/__getCatalogDirectoryContents [139810673047360] DEBUG: Obtaining the catalog contents for 1 directories: 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Discovering URL for service DataManagement/FileCatalog -> dips://dirac01.grid.hep.ph.ic.ac.uk:9197/DataManagement/FileCatalog 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to connect to: dips://dirac01.grid.hep.ph.ic.ac.uk:9197/DataManagement/FileCatalog 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Using proxy /user/agroves/.globus/x509up_u10233 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Connection timeout set to: 1 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Authenticated peer (/C=UK/O=eScience/OU=Imperial/L=Physics/CN=dirac01.grid.hep.ph.ic.ac.uk) 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: New session connecting to server at ('dirac01.grid.hep.ph.ic.ac.uk', 9197) 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Connected to: dips://dirac01.grid.hep.ph.ic.ac.uk:9197/DataManagement/FileCatalog 2020-12-18 10:31:57 UTC dirac-dms-clean-directory [139810673047360] DEBUG: New connection -> 2a0c:5bc0:c8:2:216:3eff:fe7f:1bf:9197 2020-12-18 10:32:37 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Closing socket 2020-12-18 10:32:37 UTC dirac-dms-clean-directory/DataManager/__getCatalogDirectoryContents [139810673047360] DEBUG: Found 124608 files 2020-12-18 10:32:38 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Discovering URL for service DataManagement/FileCatalog -> dips://dirac01.grid.hep.ph.ic.ac.uk:9197/DataManagement/FileCatalog 2020-12-18 10:32:38 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Trying to connect to: dips://dirac01.grid.hep.ph.ic.ac.uk:9197/DataManagement/FileCatalog 2020-12-18 10:32:38 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Using proxy /user/agroves/.globus/x509up_u10233 2020-12-18 10:32:38 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Connection timeout set to: 1 2020-12-18 10:32:38 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Authenticated peer (/C=UK/O=eScience/OU=Imperial/L=Physics/CN=dirac01.grid.hep.ph.ic.ac.uk) 2020-12-18 10:32:38 UTC dirac-dms-clean-directory [139810673047360] DEBUG: New session connecting to server at ('dirac01.grid.hep.ph.ic.ac.uk', 9197) 2020-12-18 10:32:38 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Connected to: dips://dirac01.grid.hep.ph.ic.ac.uk:9197/DataManagement/FileCatalog 2020-12-18 10:32:38 UTC dirac-dms-clean-directory [139810673047360] DEBUG: New connection -> 2a0c:5bc0:c8:2:216:3eff:fe7f:1bf:9197 2020-12-18 10:34:44 UTC dirac-dms-clean-directory [139810673047360] DEBUG: Closing socket 2020-12-18 10:34:44 UTC dirac-dms-clean-directory/DataManager/cleanLogicalDirectory [139810673047360] DEBUG: Failed to clean directory. /mu3e.org/prod/build4.4.1/BhabbaMichel FileCatalog error ( 1604 : Failed to perform isDirectory from any catalog) 2020-12-18 10:34:44 UTC dirac-dms-clean-directory [139810673047360] ERROR: Failed to clean folder '/mu3e.org/prod/build4.4.1/BhabbaMichel': FileCatalog error ( 1604 : Failed to perform isDirectory from any catalog) ________________________________ From: gridpp-dirac-users-bounces@imperial.ac.uk <gridpp-dirac-users-bounces@imperial.ac.uk> on behalf of Simon Fayer <simon.fayer05@imperial.ac.uk> Sent: 17 December 2020 17:04:17 To: gridpp-dirac-users@imperial.ac.uk Subject: Re: [Gridpp-Dirac-Users] Questions on dirac file catolog Hi Andrew, On Thu, Dec 17, 2020 at 04:46:49PM +0000, Groves, Andrew wrote:
Im trying to remove files on mass from the dirac file catolog (a whole directories worth) i've tried the command dirac-dms-clean-directory however I keep getting the error Failed to perform isDirectory from any catolog.
Im trying dirac-dms-clean-directory /mu3e.org/prod/build4.4.1.
Hmm, I think this should work (I tried it out on a small test case and verified the permissions on the directory you're trying to remove). Can you please verify that your proxy is valid and in the correct group? If the proxy looks OK, try adding the -ddd option and see if the extra debug gives you any further clues as to what the problem may be.
Also im trying to use dirac-dms-data-size, could I do this on the whole directory in one command or only on a list of files.
It can only take a list of files, but you can give it a file containing a list of files (which you can generate with another command): dirac-dms-find-lfns Path=/mu3e.org/... > files.lst dirac-dms-data-size files.lst Regards, Simon -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Hi Andrew, On Fri, Dec 18, 2020 at 11:22:11AM +0000, Groves, Andrew wrote:
Thanks for than. My proxy should be correct (I have used the same command to delete the another directory) however for this quite sizeable one is still throws a error. Ive included the debug for it as I don't understand it.
Thanks, the output doesn't really contain any clues... The "failed to perform..." error message is a fairly generic one. I've checked the matching server-side log, but there aren't any errors there.
Is there a limit on the size which it will work? There isn't a specific limit for these commands, but they will hit timeouts for very large directories, which I think may be the case here.
Can you please try generating a list of the files with dirac-dms-find-lfns and then (if that's successful) give the list file as input to the dirac-dms-remove-files command and see what happens? Regards, Simon
******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hi Simon, I am working with Andrew on mu3e grid stuff. It seems indeed that the problem with dirac-dms-clean-directory is due to a timeout. I ran the following command dirac-dms-clean-directory /mu3e.org/prod/build4.4.2/ and worked without problems , the difference being that 4.4.2 had a much smaller number of files in it. I am currently running dirac-dms-remove-files with a file containing a list of /mu3e.org/prod/build4.4.1/ files and it seems to be working. Thanks very much for your help. Cheers, Carlos ============================= Dr. Carlos A. Chavez Barajas Postdoctoral Researcher University of Liverpool, UK carlos.chavez.barajas@cern.ch carlos.chavez.barajas@gmail.com Phone: +44(0)7784546700 ============================= On Fri, Dec 18, 2020 at 1:34 PM Simon Fayer <simon.fayer05@imperial.ac.uk> wrote:
Hi Andrew,
On Fri, Dec 18, 2020 at 11:22:11AM +0000, Groves, Andrew wrote:
Thanks for than. My proxy should be correct (I have used the same command to delete the another directory) however for this quite sizeable one is still throws a error. Ive included the debug for it as I don't understand it.
Thanks, the output doesn't really contain any clues... The "failed to perform..." error message is a fairly generic one. I've checked the matching server-side log, but there aren't any errors there.
Is there a limit on the size which it will work? There isn't a specific limit for these commands, but they will hit timeouts for very large directories, which I think may be the case here.
Can you please try generating a list of the files with dirac-dms-find-lfns and then (if that's successful) give the list file as input to the dirac-dms-remove-files command and see what happens?
Regards, Simon
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
participants (3)
-
Carlos Chavez Barajas
-
Groves, Andrew
-
Simon Fayer