Issue Uploading very large single files to CA-SFU-T21-disk
******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hi grid users, I am Yongheng Xu from Lancaster and I am having trouble uploading a single large root file (114GB) to CA-SFU-T21-disk using dirac. What I did was dirac-dms-add-file /t2k.org/users/yoneheng.xu/run4a_FHC_combined.root ~/p6t_validation_samples/r4a.root CA-SFU-T21-disk -ddd However, it runs for a while (~1 hour) then fails and gives the following error message: 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: New session connecting to server at ('dirac01.grid.hep.ph.ic.ac.uk', 9133) 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: Connected to: dips://dirac01.grid.hep.ph.ic.ac.uk:9133/Accounting/DataStore 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: New connection -> 146.179.232.10:9133 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: Closing socket 2020-03-12 16:26:20 UTC dirac-dms-add-file/DataManager/putAndRegister [140363232319296] DEBUG: putAndRegister: Sending accounting took 0.0 seconds 2020-03-12 16:26:20 UTC dirac-dms-add-file/DataManager/putAndRegister [140363232319296] DEBUG: Failed to put file to Storage Element. /home/t2k/yxu/p6t_validation_samples/r4a.root: Connection timed out ( 110 : Failed to copy file /home/t2k/yxu/p6t_validation_samples/r4a.root to destination url srm://lcg-t2kse1.sfu.computecanada.ca:8443/srm/managerv2?SFN=/nd280data/t2k.org/users/yoneheng.xu/run4a_FHC_combined.root: [110] [gfalt_copy_file][perform_copy][srm_plugin_filecopy][srm_do_transfer][gfalt_copy_file][perform_copy][perform_local_copy][streamed_copy][gfal_plugin_closeG][gfal_gridftp_closeG][gfal_gridftp_closeG] Operation timed out) 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] ERROR: Error: failed to upload /t2k.org/users/yoneheng.xu/run4a_FHC_combined.root to CA-SFU-T21-disk I have tested with smaller files and they can be uploaded without any problem. Does anyone know what might be the cause of this issue? The full log is attached to this email. Many thanks, Yongheng Xu
Hi Yongheng, First of all I need to warn you that a 114GB file is going to cause you all kind of unexpected problems. Are you sure all your results need to live in one big file ? To debug your acute problem, assuming you have a CentOS7 node with cvfms enabled, you could try uploading the file with palin gfal and registering it by hand. In a *clean* (no dirac!) window do: source /cvmfs/ grid.cern.ch/umd-c7ui-test/etc/profile.d/setup-c7-ui-example.sh voms-proxy-init --voms t2k.org gfal-copy -v -T 10800 -t 10800 run4a_FHC_combined.root srm:// lcg-t2kse1.sfu.computecanada.ca:8443/srm/managerv2?SFN=/nd280data/t2k.org/users/yoneheng.xu/run4a_FHC_combined.root (The timeout is in seconds, here it's meant to be 3 h.) Let me know if that works. Regards, Daniela On Thu, 12 Mar 2020 at 20:27, Xu, Yongheng (Student) <y.xu39@lancaster.ac.uk> wrote:
This email from y.xu39@lancaster.ac.uk originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list <https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address.
Hi grid users,
I am Yongheng Xu from Lancaster and I am having trouble uploading a single large root file (114GB) to CA-SFU-T21-disk using dirac. What I did was
dirac-dms-add-file /t2k.org/users/yoneheng.xu/run4a_FHC_combined.root ~/p6t_validation_samples/r4a.root CA-SFU-T21-disk -ddd
However, it runs for a while (~1 hour) then fails and gives the following error message:
2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: New session connecting to server at ('dirac01.grid.hep.ph.ic.ac.uk', 9133)
2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: Connected to: dips:// dirac01.grid.hep.ph.ic.ac.uk:9133/Accounting/DataStore
2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: New connection -> 146.179.232.10:9133
2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: Closing socket
2020-03-12 16:26:20 UTC dirac-dms-add-file/DataManager/putAndRegister [140363232319296] DEBUG: putAndRegister: Sending accounting took 0.0 seconds
2020-03-12 16:26:20 UTC dirac-dms-add-file/DataManager/putAndRegister [140363232319296] DEBUG: Failed to put file to Storage Element. /home/t2k/yxu/p6t_validation_samples/r4a.root: Connection timed out ( 110 : Failed to copy file /home/t2k/yxu/p6t_validation_samples/r4a.root to destination url srm:// lcg-t2kse1.sfu.computecanada.ca:8443/srm/managerv2?SFN=/nd280data/t2k.org/users/yoneheng.xu/run4a_FHC_combined.root: [110] [gfalt_copy_file][perform_copy][srm_plugin_filecopy][srm_do_transfer][gfalt_copy_file][perform_copy][perform_local_copy][streamed_copy][gfal_plugin_closeG][gfal_gridftp_closeG][gfal_gridftp_closeG] Operation timed out) 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] ERROR: Error: failed to upload / t2k.org/users/yoneheng.xu/run4a_FHC_combined.root to CA-SFU-T21-disk
I have tested with smaller files and they can be uploaded without any problem. Does anyone know what might be the cause of this issue? The full log is attached to this email.
Many thanks,
Yongheng Xu -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
Hi Daniela, Thank you very much for your suggestion. I’ve talked to people working with me and we believe it indeed is necessary to exchange a single large file rather than many separate smaller files. I tried your suggestion. At beginning, it works however after some time it gave a broken pip error like: monitor: file:///home/t2k/yxu/p6t_validation_samples/r4a.root gsiftp://lcg-t2kdp1.sfu.computecanada.ca:2811//upload/2/166e64cf-eee7-4d10-8007-c93e7113eca1/run4a_FHC_combined.root 23381619 22369621 96519323648 4128 monitor: file:///home/t2k/yxu/p6t_validation_samples/r4a.root gsiftp://lcg-t2kdp1.sfu.computecanada.ca:2811//upload/2/166e64cf-eee7-4d10-8007-c93e7113eca1/run4a_FHC_combined.root 23381164 23068672 96657735680 4134 monitor: file:///home/t2k/yxu/p6t_validation_samples/r4a.root gsiftp://lcg-t2kdp1.sfu.computecanada.ca:2811//upload/2/166e64cf-eee7-4d10-8007-c93e7113eca1/run4a_FHC_combined.root 23378685 21670570 96787759104 4140 WARNING Transfer failed with: globus_xio: System error in send: Broken pipe globus_xio: A system call failed: Broken pipe gfal-copy error: 70 (Communication error on send) - globus_xio: System error in send: Broken pipe globus_xio: A system call failed: Broken pipe could this be caused be a too large file? Thanks and kind regards, Yongheng From: <gridpp-dirac-users-bounces@imperial.ac.uk> on behalf of Daniela Bauer <daniela.bauer.grid@googlemail.com> Reply to: "gridpp-dirac-users@imperial.ac.uk" <gridpp-dirac-users@imperial.ac.uk> Date: Friday, March 13, 2020 at 3:00 PM To: "gridpp-dirac-users@imperial.ac.uk" <gridpp-dirac-users@imperial.ac.uk> Subject: [External] Re: [Gridpp-Dirac-Users] Issue Uploading very large single files to CA-SFU-T21-disk This email originated outside the University. Check before clicking links or attachments. Hi Yongheng, First of all I need to warn you that a 114GB file is going to cause you all kind of unexpected problems. Are you sure all your results need to live in one big file ? To debug your acute problem, assuming you have a CentOS7 node with cvfms enabled, you could try uploading the file with palin gfal and registering it by hand. In a *clean* (no dirac!) window do: source /cvmfs/grid.cern.ch/umd-c7ui-test/etc/profile.d/setup-c7-ui-example.sh<http://grid.cern.ch/umd-c7ui-test/etc/profile.d/setup-c7-ui-example.sh> voms-proxy-init --voms t2k.org<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ft2k.org%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540159197&sdata=KTC%2FnIEEe4YdQgK1Y6m6qttVfysenkkG0KBGdhfyllg%3D&reserved=0> gfal-copy -v -T 10800 -t 10800 run4a_FHC_combined.root srm://lcg-t2kse1.sfu.computecanada.ca:8443/srm/managerv2?SFN=/nd280data/t2k.org/users/yoneheng.xu/run4a_FHC_combined.root<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flcg-t2kse1.sfu.computecanada.ca%3A8443%2Fsrm%2Fmanagerv2%3FSFN%3D%2Fnd280data%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540159197&sdata=PPovHN%2FLpr%2FSh8n5UvgXllInTIia1LWp3TTeVdyBsF0%3D&reserved=0> (The timeout is in seconds, here it's meant to be 3 h.) Let me know if that works. Regards, Daniela On Thu, 12 Mar 2020 at 20:27, Xu, Yongheng (Student) <y.xu39@lancaster.ac.uk<mailto:y.xu39@lancaster.ac.uk>> wrote: This email from y.xu39@lancaster.ac.uk<mailto:y.xu39@lancaster.ac.uk> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspam.ic.ac.uk%2FSpamConsole%2FSenders.aspx&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540159197&sdata=EQtMfewl4HGuqotiOackxlt1zLiSbk633iMzvGjuy%2Bc%3D&reserved=0> to disable email stamping for this address. Hi grid users, I am Yongheng Xu from Lancaster and I am having trouble uploading a single large root file (114GB) to CA-SFU-T21-disk using dirac. What I did was dirac-dms-add-file /t2k.org/users/yoneheng.xu/run4a_FHC_combined.root<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540169194&sdata=bqyyBXD1WZ%2Bd2MirhuLxOZHLpg%2B5XianpBlnh3AmKc4%3D&reserved=0> ~/p6t_validation_samples/r4a.root CA-SFU-T21-disk -ddd However, it runs for a while (~1 hour) then fails and gives the following error message: 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: New session connecting to server at ('dirac01.grid.hep.ph.ic.ac.uk<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdirac01.grid.hep.ph.ic.ac.uk%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540169194&sdata=4nP5aGfKqsuybOz6wsEfappfDICUpKI0SIley8VzoBY%3D&reserved=0>', 9133) 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: Connected to: dips://dirac01.grid.hep.ph.ic.ac.uk:9133/Accounting/DataStore<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdirac01.grid.hep.ph.ic.ac.uk%3A9133%2FAccounting%2FDataStore&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540179186&sdata=oOAMWkKPWbnZ9PrUmXjapfExueeM7ToqWscvrwQM9FA%3D&reserved=0> 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: New connection -> 146.179.232.10:9133<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2F146.179.232.10%3A9133%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540179186&sdata=Y8Gj89KaOESS3IGiJstvIqb2l59hE0dk4HKJdEhppNM%3D&reserved=0> 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: Closing socket 2020-03-12 16:26:20 UTC dirac-dms-add-file/DataManager/putAndRegister [140363232319296] DEBUG: putAndRegister: Sending accounting took 0.0 seconds 2020-03-12 16:26:20 UTC dirac-dms-add-file/DataManager/putAndRegister [140363232319296] DEBUG: Failed to put file to Storage Element. /home/t2k/yxu/p6t_validation_samples/r4a.root: Connection timed out ( 110 : Failed to copy file /home/t2k/yxu/p6t_validation_samples/r4a.root to destination url srm://lcg-t2kse1.sfu.computecanada.ca:8443/srm/managerv2?SFN=/nd280data/t2k.org/users/yoneheng.xu/run4a_FHC_combined.root<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flcg-t2kse1.sfu.computecanada.ca%3A8443%2Fsrm%2Fmanagerv2%3FSFN%3D%2Fnd280data%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540189180&sdata=iJl%2Bq%2BfZIRiIJwAlGK%2FLr0NXmv%2FQCpG06GfIdZxJL7Y%3D&reserved=0>: [110] [gfalt_copy_file][perform_copy][srm_plugin_filecopy][srm_do_transfer][gfalt_copy_file][perform_copy][perform_local_copy][streamed_copy][gfal_plugin_closeG][gfal_gridftp_closeG][gfal_gridftp_closeG] Operation timed out) 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] ERROR: Error: failed to upload /t2k.org/users/yoneheng.xu/run4a_FHC_combined.root<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540189180&sdata=qwazE0%2BfocGs54SozvJdGSze7xz3vvBjsycFLLShwYA%3D&reserved=0> to CA-SFU-T21-disk I have tested with smaller files and they can be uploaded without any problem. Does anyone know what might be the cause of this issue? The full log is attached to this email. Many thanks, Yongheng Xu -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk<mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.ic.ac.uk%2Fmailman%2Flistinfo%2Fgridpp-dirac-users&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540199171&sdata=xV6Orpj09wIUJWEU7XCvXbIucExf3UruNgiOYpeL8ZE%3D&reserved=0> -- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk<mailto:daniela.bauer@imperial.ac.uk> HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/<https://eur02.safelinks.protection.outlook.com/?url=http:%2F%2Fwww.hep.ph.ic.ac.uk%2F~dbauer%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637197084540199171&sdata=CDZWPcEwmOEfiIAbmPMwOvUo%2FuEh4%2FD9UVqL%2BLAJ23M%3D&reserved=0>
Hi Yongheng, yes, it might well be. I have to admit I cannot think of a single application that would have to read in a gigantic file, rather than a chain of root files... Could you please try and repeat the command but copying the to the Imperial College storage instead, so we can check whether the problem is at your or the receiving end ? At least for Imperial I can see the logs (note the triple v for extra output in the command and the path, so I can find it). gfal-copy -vvv -T 10800 -t 10800 run4a_FHC_combined.root srm:// gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/t2k/local/run4a_FHC_combined.root Regards, Daniela On Mon, 16 Mar 2020 at 12:33, Xu, Yongheng (Student) <y.xu39@lancaster.ac.uk> wrote:
Hi Daniela,
Thank you very much for your suggestion.
I’ve talked to people working with me and we believe it indeed is necessary to exchange a single large file rather than many separate smaller files.
I tried your suggestion. At beginning, it works however after some time it gave a broken pip error like:
monitor: file:///home/t2k/yxu/p6t_validation_samples/r4a.root gsiftp:// lcg-t2kdp1.sfu.computecanada.ca:2811//upload/2/166e64cf-eee7-4d10-8007-c93e7113eca1/run4a_FHC_combined.root 23381619 22369621 96519323648 4128
monitor: file:///home/t2k/yxu/p6t_validation_samples/r4a.root gsiftp:// lcg-t2kdp1.sfu.computecanada.ca:2811//upload/2/166e64cf-eee7-4d10-8007-c93e7113eca1/run4a_FHC_combined.root 23381164 23068672 96657735680 4134
monitor: file:///home/t2k/yxu/p6t_validation_samples/r4a.root gsiftp:// lcg-t2kdp1.sfu.computecanada.ca:2811//upload/2/166e64cf-eee7-4d10-8007-c93e7113eca1/run4a_FHC_combined.root 23378685 21670570 96787759104 4140
*WARNING* Transfer failed with: globus_xio: System error in send: Broken pipe globus_xio: A system call failed: Broken pipe
gfal-copy error: 70 (Communication error on send) - globus_xio: System error in send: Broken pipe globus_xio: A system call failed: Broken pipe
could this be caused be a too large file?
Thanks and kind regards,
Yongheng
*From: *<gridpp-dirac-users-bounces@imperial.ac.uk> on behalf of Daniela Bauer <daniela.bauer.grid@googlemail.com> *Reply to: *"gridpp-dirac-users@imperial.ac.uk" < gridpp-dirac-users@imperial.ac.uk> *Date: *Friday, March 13, 2020 at 3:00 PM *To: *"gridpp-dirac-users@imperial.ac.uk" < gridpp-dirac-users@imperial.ac.uk> *Subject: *[External] Re: [Gridpp-Dirac-Users] Issue Uploading very large single files to CA-SFU-T21-disk
*This email originated outside the University. Check before clicking links or attachments.*
Hi Yongheng,
First of all I need to warn you that a 114GB file is going to cause you all kind of unexpected problems. Are you sure all your results need to live in one big file ?
To debug your acute problem, assuming you have a CentOS7 node with cvfms enabled, you could try uploading the file with palin gfal and registering it by hand.
In a *clean* (no dirac!) window do:
source /cvmfs/ grid.cern.ch/umd-c7ui-test/etc/profile.d/setup-c7-ui-example.sh
voms-proxy-init --voms t2k.org <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ft2k.org%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540159197&sdata=KTC%2FnIEEe4YdQgK1Y6m6qttVfysenkkG0KBGdhfyllg%3D&reserved=0>
gfal-copy -v -T 10800 -t 10800 run4a_FHC_combined.root srm:// lcg-t2kse1.sfu.computecanada.ca:8443/srm/managerv2?SFN=/nd280data/t2k.org/users/yoneheng.xu/run4a_FHC_combined.root <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flcg-t2kse1.sfu.computecanada.ca%3A8443%2Fsrm%2Fmanagerv2%3FSFN%3D%2Fnd280data%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540159197&sdata=PPovHN%2FLpr%2FSh8n5UvgXllInTIia1LWp3TTeVdyBsF0%3D&reserved=0>
(The timeout is in seconds, here it's meant to be 3 h.)
Let me know if that works.
Regards,
Daniela
On Thu, 12 Mar 2020 at 20:27, Xu, Yongheng (Student) < y.xu39@lancaster.ac.uk> wrote:
This email from y.xu39@lancaster.ac.uk originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspam.ic.ac.uk%2FSpamConsole%2FSenders.aspx&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540159197&sdata=EQtMfewl4HGuqotiOackxlt1zLiSbk633iMzvGjuy%2Bc%3D&reserved=0> to disable email stamping for this address.
Hi grid users,
I am Yongheng Xu from Lancaster and I am having trouble uploading a single large root file (114GB) to CA-SFU-T21-disk using dirac. What I did was
dirac-dms-add-file /t2k.org/users/yoneheng.xu/run4a_FHC_combined.root <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540169194&sdata=bqyyBXD1WZ%2Bd2MirhuLxOZHLpg%2B5XianpBlnh3AmKc4%3D&reserved=0> ~/p6t_validation_samples/r4a.root CA-SFU-T21-disk -ddd
However, it runs for a while (~1 hour) then fails and gives the following error message:
2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: New session connecting to server at ('dirac01.grid.hep.ph.ic.ac.uk <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdirac01.grid.hep.ph.ic.ac.uk%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540169194&sdata=4nP5aGfKqsuybOz6wsEfappfDICUpKI0SIley8VzoBY%3D&reserved=0>', 9133)
2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: Connected to: dips:// dirac01.grid.hep.ph.ic.ac.uk:9133/Accounting/DataStore <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdirac01.grid.hep.ph.ic.ac.uk%3A9133%2FAccounting%2FDataStore&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540179186&sdata=oOAMWkKPWbnZ9PrUmXjapfExueeM7ToqWscvrwQM9FA%3D&reserved=0>
2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: New connection -> 146.179.232.10:9133 <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2F146.179.232.10%3A9133%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540179186&sdata=Y8Gj89KaOESS3IGiJstvIqb2l59hE0dk4HKJdEhppNM%3D&reserved=0>
2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: Closing socket
2020-03-12 16:26:20 UTC dirac-dms-add-file/DataManager/putAndRegister [140363232319296] DEBUG: putAndRegister: Sending accounting took 0.0 seconds
2020-03-12 16:26:20 UTC dirac-dms-add-file/DataManager/putAndRegister [140363232319296] DEBUG: Failed to put file to Storage Element. /home/t2k/yxu/p6t_validation_samples/r4a.root: Connection timed out ( 110 : Failed to copy file /home/t2k/yxu/p6t_validation_samples/r4a.root to destination url srm:// lcg-t2kse1.sfu.computecanada.ca:8443/srm/managerv2?SFN=/nd280data/t2k.org/users/yoneheng.xu/run4a_FHC_combined.root <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flcg-t2kse1.sfu.computecanada.ca%3A8443%2Fsrm%2Fmanagerv2%3FSFN%3D%2Fnd280data%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540189180&sdata=iJl%2Bq%2BfZIRiIJwAlGK%2FLr0NXmv%2FQCpG06GfIdZxJL7Y%3D&reserved=0>: [110] [gfalt_copy_file][perform_copy][srm_plugin_filecopy][srm_do_transfer][gfalt_copy_file][perform_copy][perform_local_copy][streamed_copy][gfal_plugin_closeG][gfal_gridftp_closeG][gfal_gridftp_closeG] Operation timed out) 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] ERROR: Error: failed to upload / t2k.org/users/yoneheng.xu/run4a_FHC_combined.root <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540189180&sdata=qwazE0%2BfocGs54SozvJdGSze7xz3vvBjsycFLLShwYA%3D&reserved=0> to CA-SFU-T21-disk
I have tested with smaller files and they can be uploaded without any problem. Does anyone know what might be the cause of this issue? The full log is attached to this email.
Many thanks,
Yongheng Xu
-- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.ic.ac.uk%2Fmailman%2Flistinfo%2Fgridpp-dirac-users&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540199171&sdata=xV6Orpj09wIUJWEU7XCvXbIucExf3UruNgiOYpeL8ZE%3D&reserved=0>
--
Sent from the pit of despair
----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/ <https://eur02.safelinks.protection.outlook.com/?url=http:%2F%2Fwww.hep.ph.ic.ac.uk%2F~dbauer%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637197084540199171&sdata=CDZWPcEwmOEfiIAbmPMwOvUo%2FuEh4%2FD9UVqL%2BLAJ23M%3D&reserved=0> -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
-- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/
participants (2)
-
Daniela Bauer
-
Xu, Yongheng (Student)