Hi Daniela, Thank you very much for your suggestion. I’ve talked to people working with me and we believe it indeed is necessary to exchange a single large file rather than many separate smaller files. I tried your suggestion. At beginning, it works however after some time it gave a broken pip error like: monitor: file:///home/t2k/yxu/p6t_validation_samples/r4a.root gsiftp://lcg-t2kdp1.sfu.computecanada.ca:2811//upload/2/166e64cf-eee7-4d10-8007-c93e7113eca1/run4a_FHC_combined.root 23381619 22369621 96519323648 4128 monitor: file:///home/t2k/yxu/p6t_validation_samples/r4a.root gsiftp://lcg-t2kdp1.sfu.computecanada.ca:2811//upload/2/166e64cf-eee7-4d10-8007-c93e7113eca1/run4a_FHC_combined.root 23381164 23068672 96657735680 4134 monitor: file:///home/t2k/yxu/p6t_validation_samples/r4a.root gsiftp://lcg-t2kdp1.sfu.computecanada.ca:2811//upload/2/166e64cf-eee7-4d10-8007-c93e7113eca1/run4a_FHC_combined.root 23378685 21670570 96787759104 4140 WARNING Transfer failed with: globus_xio: System error in send: Broken pipe globus_xio: A system call failed: Broken pipe gfal-copy error: 70 (Communication error on send) - globus_xio: System error in send: Broken pipe globus_xio: A system call failed: Broken pipe could this be caused be a too large file? Thanks and kind regards, Yongheng From: <gridpp-dirac-users-bounces@imperial.ac.uk> on behalf of Daniela Bauer <daniela.bauer.grid@googlemail.com> Reply to: "gridpp-dirac-users@imperial.ac.uk" <gridpp-dirac-users@imperial.ac.uk> Date: Friday, March 13, 2020 at 3:00 PM To: "gridpp-dirac-users@imperial.ac.uk" <gridpp-dirac-users@imperial.ac.uk> Subject: [External] Re: [Gridpp-Dirac-Users] Issue Uploading very large single files to CA-SFU-T21-disk This email originated outside the University. Check before clicking links or attachments. Hi Yongheng, First of all I need to warn you that a 114GB file is going to cause you all kind of unexpected problems. Are you sure all your results need to live in one big file ? To debug your acute problem, assuming you have a CentOS7 node with cvfms enabled, you could try uploading the file with palin gfal and registering it by hand. In a *clean* (no dirac!) window do: source /cvmfs/grid.cern.ch/umd-c7ui-test/etc/profile.d/setup-c7-ui-example.sh<http://grid.cern.ch/umd-c7ui-test/etc/profile.d/setup-c7-ui-example.sh> voms-proxy-init --voms t2k.org<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ft2k.org%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540159197&sdata=KTC%2FnIEEe4YdQgK1Y6m6qttVfysenkkG0KBGdhfyllg%3D&reserved=0> gfal-copy -v -T 10800 -t 10800 run4a_FHC_combined.root srm://lcg-t2kse1.sfu.computecanada.ca:8443/srm/managerv2?SFN=/nd280data/t2k.org/users/yoneheng.xu/run4a_FHC_combined.root<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flcg-t2kse1.sfu.computecanada.ca%3A8443%2Fsrm%2Fmanagerv2%3FSFN%3D%2Fnd280data%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540159197&sdata=PPovHN%2FLpr%2FSh8n5UvgXllInTIia1LWp3TTeVdyBsF0%3D&reserved=0> (The timeout is in seconds, here it's meant to be 3 h.) Let me know if that works. Regards, Daniela On Thu, 12 Mar 2020 at 20:27, Xu, Yongheng (Student) <y.xu39@lancaster.ac.uk<mailto:y.xu39@lancaster.ac.uk>> wrote: This email from y.xu39@lancaster.ac.uk<mailto:y.xu39@lancaster.ac.uk> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspam.ic.ac.uk%2FSpamConsole%2FSenders.aspx&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540159197&sdata=EQtMfewl4HGuqotiOackxlt1zLiSbk633iMzvGjuy%2Bc%3D&reserved=0> to disable email stamping for this address. Hi grid users, I am Yongheng Xu from Lancaster and I am having trouble uploading a single large root file (114GB) to CA-SFU-T21-disk using dirac. What I did was dirac-dms-add-file /t2k.org/users/yoneheng.xu/run4a_FHC_combined.root<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540169194&sdata=bqyyBXD1WZ%2Bd2MirhuLxOZHLpg%2B5XianpBlnh3AmKc4%3D&reserved=0> ~/p6t_validation_samples/r4a.root CA-SFU-T21-disk -ddd However, it runs for a while (~1 hour) then fails and gives the following error message: 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: New session connecting to server at ('dirac01.grid.hep.ph.ic.ac.uk<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdirac01.grid.hep.ph.ic.ac.uk%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540169194&sdata=4nP5aGfKqsuybOz6wsEfappfDICUpKI0SIley8VzoBY%3D&reserved=0>', 9133) 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: Connected to: dips://dirac01.grid.hep.ph.ic.ac.uk:9133/Accounting/DataStore<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdirac01.grid.hep.ph.ic.ac.uk%3A9133%2FAccounting%2FDataStore&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540179186&sdata=oOAMWkKPWbnZ9PrUmXjapfExueeM7ToqWscvrwQM9FA%3D&reserved=0> 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: New connection -> 146.179.232.10:9133<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2F146.179.232.10%3A9133%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540179186&sdata=Y8Gj89KaOESS3IGiJstvIqb2l59hE0dk4HKJdEhppNM%3D&reserved=0> 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] DEBUG: Closing socket 2020-03-12 16:26:20 UTC dirac-dms-add-file/DataManager/putAndRegister [140363232319296] DEBUG: putAndRegister: Sending accounting took 0.0 seconds 2020-03-12 16:26:20 UTC dirac-dms-add-file/DataManager/putAndRegister [140363232319296] DEBUG: Failed to put file to Storage Element. /home/t2k/yxu/p6t_validation_samples/r4a.root: Connection timed out ( 110 : Failed to copy file /home/t2k/yxu/p6t_validation_samples/r4a.root to destination url srm://lcg-t2kse1.sfu.computecanada.ca:8443/srm/managerv2?SFN=/nd280data/t2k.org/users/yoneheng.xu/run4a_FHC_combined.root<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flcg-t2kse1.sfu.computecanada.ca%3A8443%2Fsrm%2Fmanagerv2%3FSFN%3D%2Fnd280data%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540189180&sdata=iJl%2Bq%2BfZIRiIJwAlGK%2FLr0NXmv%2FQCpG06GfIdZxJL7Y%3D&reserved=0>: [110] [gfalt_copy_file][perform_copy][srm_plugin_filecopy][srm_do_transfer][gfalt_copy_file][perform_copy][perform_local_copy][streamed_copy][gfal_plugin_closeG][gfal_gridftp_closeG][gfal_gridftp_closeG] Operation timed out) 2020-03-12 16:26:20 UTC dirac-dms-add-file [140363232319296] ERROR: Error: failed to upload /t2k.org/users/yoneheng.xu/run4a_FHC_combined.root<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ft2k.org%2Fusers%2Fyoneheng.xu%2Frun4a_FHC_combined.root&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540189180&sdata=qwazE0%2BfocGs54SozvJdGSze7xz3vvBjsycFLLShwYA%3D&reserved=0> to CA-SFU-T21-disk I have tested with smaller files and they can be uploaded without any problem. Does anyone know what might be the cause of this issue? The full log is attached to this email. Many thanks, Yongheng Xu -- _______________________________________________ Gridpp-Dirac-Users mailing list Gridpp-Dirac-Users@imperial.ac.uk<mailto:Gridpp-Dirac-Users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.ic.ac.uk%2Fmailman%2Flistinfo%2Fgridpp-dirac-users&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637197084540199171&sdata=xV6Orpj09wIUJWEU7XCvXbIucExf3UruNgiOYpeL8ZE%3D&reserved=0> -- Sent from the pit of despair ----------------------------------------------------------- daniela.bauer@imperial.ac.uk<mailto:daniela.bauer@imperial.ac.uk> HEP Group/Physics Dep Imperial College London, SW7 2BW Tel: +44-(0)20-75947810 http://www.hep.ph.ic.ac.uk/~dbauer/<https://eur02.safelinks.protection.outlook.com/?url=http:%2F%2Fwww.hep.ph.ic.ac.uk%2F~dbauer%2F&data=02%7C01%7Cy.xu39%40lancaster.ac.uk%7C0217da88bde54429aab308d7c75f52fe%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637197084540199171&sdata=CDZWPcEwmOEfiIAbmPMwOvUo%2FuEh4%2FD9UVqL%2BLAJ23M%3D&reserved=0>