Re: [firedrake] Reduce number of output files for parallel runs
This is expected behaviour in Firedrake. Parallel vtu output consists of a collection of files, each containing data for a different part of the domain. It seems the most natural thing is for each process to write out data for the part of the domain it owns, otherwise we have to do extra moving of data. If we took this to the extreme of moving all the data onto one process, this would create a horrible serial bottleneck (and would be impossible if there was more data than would fit on a node). Why is the large number of files a problem? On 24 March 2017 at 08:52, Buesing, Henrik <HBuesing@eonerc.rwth-aachen.de> wrote:
Dear all,
Is it possible to reduce the number of output files in Firedrake when running in parallel? I am seeing an increase in the number of output files when increasing the number of processes used. Is this normal behavior?
Thank you!
Henrik
--
Dipl.-Math. Henrik Büsing
Institute for Applied Geophysics and Geothermal Energy
E.ON Energy Research Center
RWTH Aachen University
------------------------------------------------------
Mathieustr. 10 | Tel +49 (0)241 80 49907 <+49%20241%208049907>
52074 Aachen, Germany | Fax +49 (0)241 80 49889 <+49%20241%208049889>
------------------------------------------------------
http://www.eonerc.rwth-aachen.de/GGE
hbuesing@eonerc.rwth-aachen.de
------------------------------------------------------
Von: firedrake-bounces@imperial.ac.uk [mailto:firedrake-bounces@imperial.ac.uk] Im Auftrag von Andrew McRae Gesendet: 24 March 2017 12:01 An: firedrake@imperial.ac.uk Betreff: Re: [firedrake] Reduce number of output files for parallel runs This is expected behaviour in Firedrake. Parallel vtu output consists of a collection of files, each containing data for a different part of the domain. [Buesing, Henrik] I am just used to the single file vts output from PETSc. It seems the most natural thing is for each process to write out data for the part of the domain it owns, otherwise we have to do extra moving of data. If we took this to the extreme of moving all the data onto one process, this would create a horrible serial bottleneck (and would be impossible if there was more data than would fit on a node). [Buesing, Henrik] Very true! Why is the large number of files a problem? [Buesing, Henrik] It is not a big problem. The number of files just decreases the speed of my filesystem. E.g. when trying to access the output folder in paraview this takes some time due to the large number of files. On 24 March 2017 at 08:52, Buesing, Henrik <HBuesing@eonerc.rwth-aachen.de<mailto:HBuesing@eonerc.rwth-aachen.de>> wrote: Dear all, Is it possible to reduce the number of output files in Firedrake when running in parallel? I am seeing an increase in the number of output files when increasing the number of processes used. Is this normal behavior? Thank you! Henrik -- Dipl.-Math. Henrik Büsing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907<tel:+49%20241%208049907> 52074 Aachen, Germany | Fax +49 (0)241 80 49889<tel:+49%20241%208049889> ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing@eonerc.rwth-aachen.de<mailto:hbuesing@eonerc.rwth-aachen.de> ------------------------------------------------------
participants (2)
- 
                
                Andrew McRae
- 
                
                Buesing, Henrik