Parallel FieldConvert output in one single vtu file ?
******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hi All, I'm on Nektar version 5.1.0, ending with datasets composed of moderate size files in numbers reaching few hundreds using the Incompresible Navier-Stokes Solver, I just want to use parallel FieldConvert to speed up the conversion process (which actually provide a real gain in speed) and produce a single vtu file, since opening *.pvtu format datasets gave an error in paraview-openmpi-5.9.1 , The command I'm using for parallel FieldConvert looks like this: mpirun -n 16 FieldConvert -m mapping model.xml model_i.chk model_i.vtu this creates n .vtu files under a .pvtu header, which as I reported earlier gives rise to this error:
ERROR: In /builddir/build/BUILD/ParaView-v5.9.1/VTK/IO/XML/vtkXMLPDataReader.cxx, line 315
vtkXMLPUnstructuredGridReader (0x55ffb1431450): File for piece 0 cannot be read.
Is it possible to produce one single .vtu file out of parallel FieldConvert ? Thanks in advance, any help on this or workaround will be really appreciated, Kind regards, Saad
Ho Saad, You could try to produce a .plt file (a binary Tecplot format), and if your PV support it read this in. The benefits of plt files is size, and that it is a single file. With the PV support for plt. It used to be, that you needed to compile PV to use it, but I think now it is in the binaries (at least on Ubuntu). Cheers, Stan On 11.06.2022 11:24, Debbahi Saad wrote:
This email from saad.debbahi@gmail.com originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list <https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address.
Hi All,
I'm on Nektar version 5.1.0, ending with datasets composed of moderate size files in numbers reaching few hundreds using the Incompresible Navier-Stokes Solver, I just want to use parallel FieldConvert to speed up the conversion process (which actually provide a real gain in speed) and produce a single vtu file, since opening *.pvtu format datasets gave an error in paraview-openmpi-5.9.1 , The command I'm using for parallel FieldConvert looks like this:
mpirun -n 16 FieldConvert -m mapping model.xml model_i.chk model_i.vtu
this creates n .vtu files under a .pvtu header, which as I reported earlier gives rise to this error:
ERROR: In /builddir/build/BUILD/ParaView-v5.9.1/VTK/IO/XML/vtkXMLPDataReader.cxx, line 315
vtkXMLPUnstructuredGridReader (0x55ffb1431450): File for piece 0 cannot be read.
Is it possible to produce one single .vtu file out of parallel FieldConvert ?
Thanks in advance, any help on this or workaround will be really appreciated,
Kind regards,
Saad
_______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
-- Best regards, Stanisław Gepner, Ph.D. *Faculty of Power and Aeronautical Engineering* Nowowiejska 21/25 Str. 00-665 Warsaw, Poland phone +48 (22) 234 51 70 <https://www.pw.edu.pl/engpw>
Hi Saad, Actually, I had the same problem but I may say the problem was solved by using FieldConvert in serial, otherwise by using IncNavierStokesSolver in parallel. I mean, one can obtain a single vtu (or plt) file, even if there are multiple chk (or fld) "files in folders" generated in parallel FieldConvert, by converting multiple files into a single output file by applying a "one-time-only" serial run. However, it should be noted that this makes the process longer, but may help your problem. (e.g., I am working with more than 250 processors, producing big data, using this method to get results and it works.) Regards, Selman BaysalR. A. in HydraulicsDept. of Civil Eng'gÇanakkale Onsekiz Mart University--ITU Civil Eng., BSc.ITU Coastal Sci. & Eng., MSc.GSM: +905436180777 Kimden: stanislaw.gepner@pw.edu.pl;Kime: nektar-users@imperial.ac.uk; - 11/06/22 - 05:49Konu: Re: [Nektar-users] Parallel FieldConvert output in one single vtu file ? Ho Saad, You could try to produce a .plt file (a binary Tecplot format), and if your PV support it read this in. The benefits of plt files is size, and that it is a single file. With the PV support for plt. It used to be, that you needed to compile PV to use it, but I think now it is in the binaries (at least on Ubuntu). Cheers, Stan On 11.06.2022 11:24, Debbahi Saad wrote: This email from saad.debbahi@gmail.com originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list to disable email stamping for this address. Hi All, I'm on Nektar version 5.1.0, ending with datasets composed of moderate size files in numbers reaching few hundreds using the Incompresible Navier-Stokes Solver, I just want to use parallel FieldConvert to speed up the conversion process (which actually provide a real gain in speed) and produce a single vtu file, since opening *.pvtu format datasets gave an error in paraview-openmpi-5.9.1 , The command I'm using for parallel FieldConvert looks like this: mpirun -n 16 FieldConvert -m mapping model.xml model_i.chk model_i.vtu this creates n .vtu files under a .pvtu header, which as I reported earlier gives rise to this error: ERROR: In /builddir/build/BUILD/ParaView-v5.9.1/VTK/IO/XML/vtkXMLPDataReader.cxx, line 315 vtkXMLPUnstructuredGridReader (0x55ffb1431450): File for piece 0 cannot be read. Is it possible to produce one single .vtu file out of parallel FieldConvert ? Thanks in advance, any help on this or workaround will be really appreciated, Kind regards, Saad _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/nektar-users -- Best regards, Stanisław Gepner, Ph.D. Faculty of Power and Aeronautical Engineering Nowowiejska 21/25 Str. 00-665 Warsaw, Poland phone +48 (22) 234 51 70 [image : ]
Hi, Sorry for the delayed replay, Indeed, running the Incompressible N-S solver in parallel does generate partitioned data, what would be ideal for me is going through the parallel solver .chk files using parallel I/O within paraview for speed. At this point, I think it is necessary to convert the data to vtu, dat or plt for paraview to handle it. I'm running the Incompressible navier-stokes solver with the "--io-format Hdf5" command line option as suggested in the user-guide, and converting the data set to tecplot .dat file as suggested by Stan, using a shell script with a loop, while testing .plt format inside paraview, plus, I have to use a paraview calculator to include the mapping on the geometry. Please, I would like to know if it is possible to directly use the parallel data output from the IncNavierStokesSolver in paraview or is it really mandatory to convert the dataset ? Thank you, regards, Saad Le lun. 13 juin 2022 à 10:58, SELMAN BAYSAL <selmanbaysal@comu.edu.tr> a écrit :
Hi Saad,
Actually, I had the same problem but I may say the problem was solved by using FieldConvert in serial, otherwise by using IncNavierStokesSolver in parallel. I mean, one can obtain a single vtu (or plt) file, even if there are multiple chk (or fld) "files in folders" generated in parallel FieldConvert, by converting multiple files into a single output file by applying a "one-time-only" serial run.
However, it should be noted that this makes the process longer, but may help your problem. (e.g., I am working with more than 250 processors, producing big data, using this method to get results and it works.)
Regards,
Selman Baysal R. A. in Hydraulics Dept. of Civil Eng'g Çanakkale Onsekiz Mart University -- ITU Civil Eng., BSc. ITU Coastal Sci. & Eng., MSc. GSM: +905436180777
------------------------------ *Kimden:* stanislaw.gepner@pw.edu.pl; *Kime:* nektar-users@imperial.ac.uk; - 11/06/22 - 05:49 *Konu:* Re: [Nektar-users] Parallel FieldConvert output in one single vtu file ?
Ho Saad,
You could try to produce a .plt file (a binary Tecplot format), and if your PV support it read this in. The benefits of plt files is size, and that it is a single file. With the PV support for plt. It used to be, that you needed to compile PV to use it, but I think now it is in the binaries (at least on Ubuntu).
Cheers, Stan On 11.06.2022 11:24, Debbahi Saad wrote:
This email from saad.debbahi@gmail.com originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list <https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address.
Hi All,
I'm on Nektar version 5.1.0, ending with datasets composed of moderate size files in numbers reaching few hundreds using the Incompresible Navier-Stokes Solver, I just want to use parallel FieldConvert to speed up the conversion process (which actually provide a real gain in speed) and produce a single vtu file, since opening *.pvtu format datasets gave an error in paraview-openmpi-5.9.1 , The command I'm using for parallel FieldConvert looks like this:
mpirun -n 16 FieldConvert -m mapping model.xml model_i.chk model_i.vtu
this creates n .vtu files under a .pvtu header, which as I reported earlier gives rise to this error:
ERROR: In /builddir/build/BUILD/ParaView-v5.9.1/VTK/IO/XML/vtkXMLPDataReader.cxx, line 315
vtkXMLPUnstructuredGridReader (0x55ffb1431450): File for piece 0 cannot be read.
Is it possible to produce one single .vtu file out of parallel FieldConvert ?
Thanks in advance, any help on this or workaround will be really appreciated,
Kind regards,
Saad
_______________________________________________ Nektar-users mailing listNektar-users@imperial.ac.ukhttps://mailman.ic.ac.uk/mailman/listinfo/nektar-users
-- Best regards, Stanisław Gepner, Ph.D.
*Faculty of Power and Aeronautical Engineering* Nowowiejska 21/25 Str. 00-665 Warsaw, Poland phone +48 (22) 234 51 70
[image : ] <https://www.pw.edu.pl/engpw> _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
HI Saad, Currently it is necessary to convert the dataset since we do not have a direct read module. There have been some efforts in the past for an old version on Nektar but we do not have anything for Nektar++. Sorry, Spencer. On 15 Jun 2022, at 12:50, Debbahi Saad <saad.debbahi@gmail.com<mailto:saad.debbahi@gmail.com>> wrote: Hi, Sorry for the delayed replay, Indeed, running the Incompressible N-S solver in parallel does generate partitioned data, what would be ideal for me is going through the parallel solver .chk files using parallel I/O within paraview for speed. At this point, I think it is necessary to convert the data to vtu, dat or plt for paraview to handle it. I'm running the Incompressible navier-stokes solver with the "--io-format Hdf5" command line option as suggested in the user-guide, and converting the data set to tecplot .dat file as suggested by Stan, using a shell script with a loop, while testing .plt format inside paraview, plus, I have to use a paraview calculator to include the mapping on the geometry. Please, I would like to know if it is possible to directly use the parallel data output from the IncNavierStokesSolver in paraview or is it really mandatory to convert the dataset ? Thank you, regards, Saad Le lun. 13 juin 2022 à 10:58, SELMAN BAYSAL <selmanbaysal@comu.edu.tr<mailto:selmanbaysal@comu.edu.tr>> a écrit : Hi Saad, Actually, I had the same problem but I may say the problem was solved by using FieldConvert in serial, otherwise by using IncNavierStokesSolver in parallel. I mean, one can obtain a single vtu (or plt) file, even if there are multiple chk (or fld) "files in folders" generated in parallel FieldConvert, by converting multiple files into a single output file by applying a "one-time-only" serial run. However, it should be noted that this makes the process longer, but may help your problem. (e.g., I am working with more than 250 processors, producing big data, using this method to get results and it works.) Regards, Selman Baysal R. A. in Hydraulics Dept. of Civil Eng'g Çanakkale Onsekiz Mart University -- ITU Civil Eng., BSc. ITU Coastal Sci. & Eng., MSc. GSM: +905436180777 ________________________________ Kimden: stanislaw.gepner@pw.edu.pl<mailto:stanislaw.gepner@pw.edu.pl>; Kime: nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>; - 11/06/22 - 05:49 Konu: Re: [Nektar-users] Parallel FieldConvert output in one single vtu file ? Ho Saad, You could try to produce a .plt file (a binary Tecplot format), and if your PV support it read this in. The benefits of plt files is size, and that it is a single file. With the PV support for plt. It used to be, that you needed to compile PV to use it, but I think now it is in the binaries (at least on Ubuntu). Cheers, Stan On 11.06.2022 11:24, Debbahi Saad wrote: This email from saad.debbahi@gmail.com<mailto:saad.debbahi@gmail.com> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hi All, I'm on Nektar version 5.1.0, ending with datasets composed of moderate size files in numbers reaching few hundreds using the Incompresible Navier-Stokes Solver, I just want to use parallel FieldConvert to speed up the conversion process (which actually provide a real gain in speed) and produce a single vtu file, since opening *.pvtu format datasets gave an error in paraview-openmpi-5.9.1 , The command I'm using for parallel FieldConvert looks like this: mpirun -n 16 FieldConvert -m mapping model.xml model_i.chk model_i.vtu this creates n .vtu files under a .pvtu header, which as I reported earlier gives rise to this error: ERROR: In /builddir/build/BUILD/ParaView-v5.9.1/VTK/IO/XML/vtkXMLPDataReader.cxx, line 315 vtkXMLPUnstructuredGridReader (0x55ffb1431450): File for piece 0 cannot be read. Is it possible to produce one single .vtu file out of parallel FieldConvert ? Thanks in advance, any help on this or workaround will be really appreciated, Kind regards, Saad _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users -- Best regards, Stanisław Gepner, Ph.D. Faculty of Power and Aeronautical Engineering Nowowiejska 21/25 Str. 00-665 Warsaw, Poland phone +48 (22) 234 51 70 [image : ] <https://www.pw.edu.pl/engpw> _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
Hi Spencer, The speed-up using "mpirun -np FieldConvert" to tecplot (.dat) is already fairly substantial, I'll keep up with the Hdf5 output flag and convert subsequently, I have to say I'm really happy with the x4 to x6 speed-up on parallel FieldConvert, It was just a question out of curiosity. Thank you, Regards, Saad Le mer. 15 juin 2022 à 14:14, Sherwin, Spencer J <s.sherwin@imperial.ac.uk> a écrit :
HI Saad,
Currently it is necessary to convert the dataset since we do not have a direct read module. There have been some efforts in the past for an old version on Nektar but we do not have anything for Nektar++.
Sorry, Spencer.
On 15 Jun 2022, at 12:50, Debbahi Saad <saad.debbahi@gmail.com> wrote:
Hi,
Sorry for the delayed replay, Indeed, running the Incompressible N-S solver in parallel does generate partitioned data, what would be ideal for me is going through the parallel solver .chk files using parallel I/O within paraview for speed. At this point, I think it is necessary to convert the data to vtu, dat or plt for paraview to handle it.
I'm running the Incompressible navier-stokes solver with the "--io-format Hdf5" command line option as suggested in the user-guide, and converting the data set to tecplot .dat file as suggested by Stan, using a shell script with a loop, while testing .plt format inside paraview, plus, I have to use a paraview calculator to include the mapping on the geometry.
Please, I would like to know if it is possible to directly use the parallel data output from the IncNavierStokesSolver in paraview or is it really mandatory to convert the dataset ?
Thank you,
regards,
Saad
Le lun. 13 juin 2022 à 10:58, SELMAN BAYSAL <selmanbaysal@comu.edu.tr> a écrit :
Hi Saad,
Actually, I had the same problem but I may say the problem was solved by using FieldConvert in serial, otherwise by using IncNavierStokesSolver in parallel. I mean, one can obtain a single vtu (or plt) file, even if there are multiple chk (or fld) "files in folders" generated in parallel FieldConvert, by converting multiple files into a single output file by applying a "one-time-only" serial run.
However, it should be noted that this makes the process longer, but may help your problem. (e.g., I am working with more than 250 processors, producing big data, using this method to get results and it works.)
Regards,
Selman Baysal R. A. in Hydraulics Dept. of Civil Eng'g Çanakkale Onsekiz Mart University -- ITU Civil Eng., BSc. ITU Coastal Sci. & Eng., MSc. GSM: +905436180777
------------------------------ *Kimden:* stanislaw.gepner@pw.edu.pl; *Kime:* nektar-users@imperial.ac.uk; - 11/06/22 - 05:49 *Konu:* Re: [Nektar-users] Parallel FieldConvert output in one single vtu file ?
Ho Saad,
You could try to produce a .plt file (a binary Tecplot format), and if your PV support it read this in. The benefits of plt files is size, and that it is a single file. With the PV support for plt. It used to be, that you needed to compile PV to use it, but I think now it is in the binaries (at least on Ubuntu).
Cheers, Stan On 11.06.2022 11:24, Debbahi Saad wrote:
This email from saad.debbahi@gmail.com originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list <https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address.
Hi All,
I'm on Nektar version 5.1.0, ending with datasets composed of moderate size files in numbers reaching few hundreds using the Incompresible Navier-Stokes Solver, I just want to use parallel FieldConvert to speed up the conversion process (which actually provide a real gain in speed) and produce a single vtu file, since opening *.pvtu format datasets gave an error in paraview-openmpi-5.9.1 , The command I'm using for parallel FieldConvert looks like this:
mpirun -n 16 FieldConvert -m mapping model.xml model_i.chk model_i.vtu
this creates n .vtu files under a .pvtu header, which as I reported earlier gives rise to this error:
ERROR: In /builddir/build/BUILD/ParaView-v5.9.1/VTK/IO/XML/vtkXMLPDataReader.cxx, line 315 vtkXMLPUnstructuredGridReader (0x55ffb1431450): File for piece 0 cannot be read.
Is it possible to produce one single .vtu file out of parallel FieldConvert ?
Thanks in advance, any help on this or workaround will be really appreciated,
Kind regards,
Saad
_______________________________________________ Nektar-users mailing listNektar-users@imperial.ac.ukhttps://mailman.ic.ac.uk/mailman/listinfo/nektar-users
-- Best regards, Stanisław Gepner, Ph.D.
*Faculty of Power and Aeronautical Engineering* Nowowiejska 21/25 Str. 00-665 Warsaw, Poland phone +48 (22) 234 51 70
[image : ] <https://www.pw.edu.pl/engpw> _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
_______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
participants (4)
-
Debbahi Saad
-
SELMAN BAYSAL
-
Sherwin, Spencer J
-
Stanislaw Gepner