Optimisation Issues
******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac
Hi Isaac, I noticed that your optimization file session.opt is basically empty: <?xml version="1.0" encoding="utf-8" ?> <NEKTAR> <COLLECTIONS /> </NEKTAR> So, my understanding is that in this case, no optimization is going to be performed at the beginning of your run. You can check if nektar++ is performing an optimization at the beginning of your run by using the -verbose option srun IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm --verbose I would simply recommend deleting your empty session.opt file and try again. If a session.opt already exist, nektar++ will simply read the existing optfile. You can also specify the name of the optfile to use by using the -useoptfile option as in srun IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm --useoptfile optfile.opt Best, Jacques From: nektar-users-bounces@imperial.ac.uk <nektar-users-bounces@imperial.ac.uk> On Behalf Of Isaac Rosin Sent: Wednesday, August 9, 2023 2:09 AM To: nektar-users@imperial.ac.uk Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca> Subject: [Nektar-users] Optimisation Issues This email from isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac
Hi Isaac I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run. Could you let me know if it had a significant influence? Kind regards syavash On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca> wrote:
This email from isaac.rosin1@ucalgary.ca originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list <https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address.
Hello Nektar Community,
I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters.
I look forward to hearing from you!
All the best, Isaac _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
Hi Ehsan, The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”. Best, Jacques From: nektar-users-bounces@imperial.ac.uk <nektar-users-bounces@imperial.ac.uk> On Behalf Of Ehsan Asgari Sent: Wednesday, August 9, 2023 3:23 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk Subject: Re: [Nektar-users] Optimisation Issues Hi Isaac I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run. Could you let me know if it had a significant influence? Kind regards syavash On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> wrote: This email from isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
Thank you Jacques Here is the .opt file generated: <NEKTAR>
<COLLECTIONS> <OPERATOR TYPE="BwdTrans"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="Helmholtz"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="NoCollection" /> </OPERATOR> <OPERATOR TYPE="IProductWRTBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="IProductWRTDerivBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="PhysDeriv"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="MatrixFree" /> </OPERATOR> </COLLECTIONS> </NEKTAR>
I had manually added <COLLECTIONS DEFAULT="auto" /> to my session file. Are the above results of the latter? Kind regards syavash On Wed, Aug 9, 2023 at 6:13 PM Jacques Xing <jacques.xing@kcl.ac.uk> wrote:
Hi Ehsan,
The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”.
Best,
Jacques
*From:* nektar-users-bounces@imperial.ac.uk < nektar-users-bounces@imperial.ac.uk> *On Behalf Of *Ehsan Asgari *Sent:* Wednesday, August 9, 2023 3:23 PM *To:* Isaac Rosin <isaac.rosin1@ucalgary.ca> *Cc:* Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk *Subject:* Re: [Nektar-users] Optimisation Issues
Hi Isaac
I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run.
Could you let me know if it had a significant influence?
Kind regards
syavash
On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca> wrote:
This email from isaac.rosin1@ucalgary.ca originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list <https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address.
Hello Nektar Community,
I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters.
I look forward to hearing from you!
All the best,
Isaac
_______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
Hi all, I forgot to remove the .opt file from the folder I sent. The issue is not with the collections. I have experimented with them and found that starting with no .opt file when running a simulation and letting Nektar++ create it automatically gives the best results. I always do this now, but the gains are not significant enough. Here is the drive folder again: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... It may be that this is not an optimisation issue but rather a debugging one. I feel that the issue is deeper than collections. I ran the same case on very similar meshes in OpenFOAM and it was also slower than it should have been. Would any of you be able to run this briefly on your own machines and tell me: 1) how long it took to save three or so time steps and 2) how many cores you ran on? If you are unable to do this, do you know of any other optimization/debugging techniques? Thank you very much, Isaac ________________________________ From: Ehsan Asgari <eh.asgari@gmail.com> Sent: Wednesday, August 9, 2023 10:42 AM To: Jacques Xing <jacques.xing@kcl.ac.uk> Cc: Isaac Rosin <isaac.rosin1@ucalgary.ca>; Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk <nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues [△EXTERNAL] Thank you Jacques Here is the .opt file generated: <NEKTAR> <COLLECTIONS> <OPERATOR TYPE="BwdTrans"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="Helmholtz"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="NoCollection" /> </OPERATOR> <OPERATOR TYPE="IProductWRTBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="IProductWRTDerivBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="PhysDeriv"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="MatrixFree" /> </OPERATOR> </COLLECTIONS> </NEKTAR> I had manually added <COLLECTIONS DEFAULT="auto" /> to my session file. Are the above results of the latter? Kind regards syavash On Wed, Aug 9, 2023 at 6:13 PM Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> wrote: Hi Ehsan, The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”. Best, Jacques From: nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk> <nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk>> On Behalf Of Ehsan Asgari Sent: Wednesday, August 9, 2023 3:23 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Isaac I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run. Could you let me know if it had a significant influence? Kind regards syavash On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> wrote: This email from isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
Hi Isaac, Did you compile Nektar++ in release or debug mode? Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca> Sent: Wednesday, August 9, 2023 7:36 PM To: Ehsan Asgari <eh.asgari@gmail.com>; Jacques Xing <jacques.xing@kcl.ac.uk> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk Subject: Re: [Nektar-users] Optimisation Issues Hi all, I forgot to remove the .opt file from the folder I sent. The issue is not with the collections. I have experimented with them and found that starting with no .opt file when running a simulation and letting Nektar++ create it automatically gives the best results. I always do this now, but the gains are not significant enough. Here is the drive folder again: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... It may be that this is not an optimisation issue but rather a debugging one. I feel that the issue is deeper than collections. I ran the same case on very similar meshes in OpenFOAM and it was also slower than it should have been. Would any of you be able to run this briefly on your own machines and tell me: 1) how long it took to save three or so time steps and 2) how many cores you ran on? If you are unable to do this, do you know of any other optimization/debugging techniques? Thank you very much, Isaac ________________________________ From: Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Sent: Wednesday, August 9, 2023 10:42 AM To: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: Re: [Nektar-users] Optimisation Issues [△EXTERNAL] Thank you Jacques Here is the .opt file generated: <NEKTAR> <COLLECTIONS> <OPERATOR TYPE="BwdTrans"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="Helmholtz"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="NoCollection" /> </OPERATOR> <OPERATOR TYPE="IProductWRTBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="IProductWRTDerivBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="PhysDeriv"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="MatrixFree" /> </OPERATOR> </COLLECTIONS> </NEKTAR> I had manually added <COLLECTIONS DEFAULT="auto" /> to my session file. Are the above results of the latter? Kind regards syavash On Wed, Aug 9, 2023 at 6:13 PM Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> wrote: Hi Ehsan, The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”. Best, Jacques From: nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk> <nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk>> On Behalf Of Ehsan Asgari Sent: Wednesday, August 9, 2023 3:23 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Isaac I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run. Could you let me know if it had a significant influence? Kind regards syavash On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> wrote: This email from isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
Hi Jacques, In release mode. Regards, Isaac ________________________________ From: Jacques Xing <jacques.xing@kcl.ac.uk> Sent: Wednesday, August 9, 2023 12:41 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca>; Ehsan Asgari <eh.asgari@gmail.com> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk <nektar-users@imperial.ac.uk> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, Did you compile Nektar++ in release or debug mode? Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca> Sent: Wednesday, August 9, 2023 7:36 PM To: Ehsan Asgari <eh.asgari@gmail.com>; Jacques Xing <jacques.xing@kcl.ac.uk> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk Subject: Re: [Nektar-users] Optimisation Issues Hi all, I forgot to remove the .opt file from the folder I sent. The issue is not with the collections. I have experimented with them and found that starting with no .opt file when running a simulation and letting Nektar++ create it automatically gives the best results. I always do this now, but the gains are not significant enough. Here is the drive folder again: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... It may be that this is not an optimisation issue but rather a debugging one. I feel that the issue is deeper than collections. I ran the same case on very similar meshes in OpenFOAM and it was also slower than it should have been. Would any of you be able to run this briefly on your own machines and tell me: 1) how long it took to save three or so time steps and 2) how many cores you ran on? If you are unable to do this, do you know of any other optimization/debugging techniques? Thank you very much, Isaac ________________________________ From: Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Sent: Wednesday, August 9, 2023 10:42 AM To: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: Re: [Nektar-users] Optimisation Issues [△EXTERNAL] Thank you Jacques Here is the .opt file generated: <NEKTAR> <COLLECTIONS> <OPERATOR TYPE="BwdTrans"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="Helmholtz"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="NoCollection" /> </OPERATOR> <OPERATOR TYPE="IProductWRTBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="IProductWRTDerivBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="PhysDeriv"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="MatrixFree" /> </OPERATOR> </COLLECTIONS> </NEKTAR> I had manually added <COLLECTIONS DEFAULT="auto" /> to my session file. Are the above results of the latter? Kind regards syavash On Wed, Aug 9, 2023 at 6:13 PM Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> wrote: Hi Ehsan, The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”. Best, Jacques From: nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk> <nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk>> On Behalf Of Ehsan Asgari Sent: Wednesday, August 9, 2023 3:23 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Isaac I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run. Could you let me know if it had a significant influence? Kind regards syavash On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> wrote: This email from isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
Hi Isaac, I am not sure why your simulation is so slow but providing a log file could be helpful: srun IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm > outfile.log Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca> Sent: Wednesday, August 9, 2023 7:44 PM To: Jacques Xing <jacques.xing@kcl.ac.uk>; Ehsan Asgari <eh.asgari@gmail.com> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk Subject: Re: [Nektar-users] Optimisation Issues Hi Jacques, In release mode. Regards, Isaac ________________________________ From: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Sent: Wednesday, August 9, 2023 12:41 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, Did you compile Nektar++ in release or debug mode? Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Sent: Wednesday, August 9, 2023 7:36 PM To: Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>>; Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi all, I forgot to remove the .opt file from the folder I sent. The issue is not with the collections. I have experimented with them and found that starting with no .opt file when running a simulation and letting Nektar++ create it automatically gives the best results. I always do this now, but the gains are not significant enough. Here is the drive folder again: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... It may be that this is not an optimisation issue but rather a debugging one. I feel that the issue is deeper than collections. I ran the same case on very similar meshes in OpenFOAM and it was also slower than it should have been. Would any of you be able to run this briefly on your own machines and tell me: 1) how long it took to save three or so time steps and 2) how many cores you ran on? If you are unable to do this, do you know of any other optimization/debugging techniques? Thank you very much, Isaac ________________________________ From: Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Sent: Wednesday, August 9, 2023 10:42 AM To: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: Re: [Nektar-users] Optimisation Issues [△EXTERNAL] Thank you Jacques Here is the .opt file generated: <NEKTAR> <COLLECTIONS> <OPERATOR TYPE="BwdTrans"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="Helmholtz"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="NoCollection" /> </OPERATOR> <OPERATOR TYPE="IProductWRTBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="IProductWRTDerivBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="PhysDeriv"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="MatrixFree" /> </OPERATOR> </COLLECTIONS> </NEKTAR> I had manually added <COLLECTIONS DEFAULT="auto" /> to my session file. Are the above results of the latter? Kind regards syavash On Wed, Aug 9, 2023 at 6:13 PM Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> wrote: Hi Ehsan, The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”. Best, Jacques From: nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk> <nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk>> On Behalf Of Ehsan Asgari Sent: Wednesday, August 9, 2023 3:23 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Isaac I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run. Could you let me know if it had a significant influence? Kind regards syavash On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> wrote: This email from isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
Hi Jacques, Here is a log file produced by running the command: mpirun -np 6 IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm -v > outfile.log I know that --use-hdf5-node-comm is supposed to use a lot of memory but it seems like it was using too much for my case. When producing this log file, by the time I stopped the simulation, it was using 30GB of RAM and still steadily increasing. I have had crashes on clusters before by exceeding the memory limit with similar cases to this. I remedied it those other times by coarsening the mesh which how I arrived at the current mesh. I can't afford to coarsen the mesh further. Perhaps this issue with the memory is a clue to finding the problem. Thank you, Isaac ________________________________ From: Jacques Xing <jacques.xing@kcl.ac.uk> Sent: Wednesday, August 9, 2023 1:34 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca>; Ehsan Asgari <eh.asgari@gmail.com> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk <nektar-users@imperial.ac.uk> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, I am not sure why your simulation is so slow but providing a log file could be helpful: srun IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm > outfile.log Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca> Sent: Wednesday, August 9, 2023 7:44 PM To: Jacques Xing <jacques.xing@kcl.ac.uk>; Ehsan Asgari <eh.asgari@gmail.com> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk Subject: Re: [Nektar-users] Optimisation Issues Hi Jacques, In release mode. Regards, Isaac ________________________________ From: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Sent: Wednesday, August 9, 2023 12:41 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, Did you compile Nektar++ in release or debug mode? Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Sent: Wednesday, August 9, 2023 7:36 PM To: Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>>; Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi all, I forgot to remove the .opt file from the folder I sent. The issue is not with the collections. I have experimented with them and found that starting with no .opt file when running a simulation and letting Nektar++ create it automatically gives the best results. I always do this now, but the gains are not significant enough. Here is the drive folder again: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... It may be that this is not an optimisation issue but rather a debugging one. I feel that the issue is deeper than collections. I ran the same case on very similar meshes in OpenFOAM and it was also slower than it should have been. Would any of you be able to run this briefly on your own machines and tell me: 1) how long it took to save three or so time steps and 2) how many cores you ran on? If you are unable to do this, do you know of any other optimization/debugging techniques? Thank you very much, Isaac ________________________________ From: Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Sent: Wednesday, August 9, 2023 10:42 AM To: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: Re: [Nektar-users] Optimisation Issues [△EXTERNAL] Thank you Jacques Here is the .opt file generated: <NEKTAR> <COLLECTIONS> <OPERATOR TYPE="BwdTrans"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="Helmholtz"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="NoCollection" /> </OPERATOR> <OPERATOR TYPE="IProductWRTBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="IProductWRTDerivBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="PhysDeriv"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="MatrixFree" /> </OPERATOR> </COLLECTIONS> </NEKTAR> I had manually added <COLLECTIONS DEFAULT="auto" /> to my session file. Are the above results of the latter? Kind regards syavash On Wed, Aug 9, 2023 at 6:13 PM Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> wrote: Hi Ehsan, The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”. Best, Jacques From: nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk> <nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk>> On Behalf Of Ehsan Asgari Sent: Wednesday, August 9, 2023 3:23 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Isaac I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run. Could you let me know if it had a significant influence? Kind regards syavash On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> wrote: This email from isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
Hi Isaac, Memory issues might be a problem. However, I took a quick look at the log file and I would also point out that the number of CG (conjugate gradient) iterations can be quite large. Some improvement might be achieved by changing the parameters of the linear system solvers, specified within the following tag <GLOBALSYSSOLNINFO>. I refer you to the documentation for more details. Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca> Sent: Wednesday, August 9, 2023 9:26 PM To: Jacques Xing <jacques.xing@kcl.ac.uk>; Ehsan Asgari <eh.asgari@gmail.com> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk Subject: Re: [Nektar-users] Optimisation Issues Hi Jacques, Here is a log file produced by running the command: mpirun -np 6 IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm -v > outfile.log I know that --use-hdf5-node-comm is supposed to use a lot of memory but it seems like it was using too much for my case. When producing this log file, by the time I stopped the simulation, it was using 30GB of RAM and still steadily increasing. I have had crashes on clusters before by exceeding the memory limit with similar cases to this. I remedied it those other times by coarsening the mesh which how I arrived at the current mesh. I can't afford to coarsen the mesh further. Perhaps this issue with the memory is a clue to finding the problem. Thank you, Isaac ________________________________ From: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Sent: Wednesday, August 9, 2023 1:34 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, I am not sure why your simulation is so slow but providing a log file could be helpful: srun IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm > outfile.log Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Sent: Wednesday, August 9, 2023 7:44 PM To: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>>; Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Jacques, In release mode. Regards, Isaac ________________________________ From: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Sent: Wednesday, August 9, 2023 12:41 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, Did you compile Nektar++ in release or debug mode? Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Sent: Wednesday, August 9, 2023 7:36 PM To: Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>>; Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi all, I forgot to remove the .opt file from the folder I sent. The issue is not with the collections. I have experimented with them and found that starting with no .opt file when running a simulation and letting Nektar++ create it automatically gives the best results. I always do this now, but the gains are not significant enough. Here is the drive folder again: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... It may be that this is not an optimisation issue but rather a debugging one. I feel that the issue is deeper than collections. I ran the same case on very similar meshes in OpenFOAM and it was also slower than it should have been. Would any of you be able to run this briefly on your own machines and tell me: 1) how long it took to save three or so time steps and 2) how many cores you ran on? If you are unable to do this, do you know of any other optimization/debugging techniques? Thank you very much, Isaac ________________________________ From: Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Sent: Wednesday, August 9, 2023 10:42 AM To: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: Re: [Nektar-users] Optimisation Issues [△EXTERNAL] Thank you Jacques Here is the .opt file generated: <NEKTAR> <COLLECTIONS> <OPERATOR TYPE="BwdTrans"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="Helmholtz"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="NoCollection" /> </OPERATOR> <OPERATOR TYPE="IProductWRTBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="IProductWRTDerivBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="PhysDeriv"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="MatrixFree" /> </OPERATOR> </COLLECTIONS> </NEKTAR> I had manually added <COLLECTIONS DEFAULT="auto" /> to my session file. Are the above results of the latter? Kind regards syavash On Wed, Aug 9, 2023 at 6:13 PM Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> wrote: Hi Ehsan, The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”. Best, Jacques From: nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk> <nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk>> On Behalf Of Ehsan Asgari Sent: Wednesday, August 9, 2023 3:23 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Isaac I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run. Could you let me know if it had a significant influence? Kind regards syavash On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> wrote: This email from isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
Hi Jacques, I will investigate the solver info. Do you have any ideas about why the simulation may be using so much memory? Could it have anything to do with my mesh? Before I wasn't selecting the Optimize 3D option in gmsh. After trying it, I haven't noticed any improvements, but perhaps there are other areas of the mesh preparation I could look at? Thank you for your help so far. Best, Isaac ________________________________ From: Jacques Xing <jacques.xing@kcl.ac.uk> Sent: Wednesday, August 9, 2023 2:51 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca>; Ehsan Asgari <eh.asgari@gmail.com> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk <nektar-users@imperial.ac.uk> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, Memory issues might be a problem. However, I took a quick look at the log file and I would also point out that the number of CG (conjugate gradient) iterations can be quite large. Some improvement might be achieved by changing the parameters of the linear system solvers, specified within the following tag <GLOBALSYSSOLNINFO>. I refer you to the documentation for more details. Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca> Sent: Wednesday, August 9, 2023 9:26 PM To: Jacques Xing <jacques.xing@kcl.ac.uk>; Ehsan Asgari <eh.asgari@gmail.com> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk Subject: Re: [Nektar-users] Optimisation Issues Hi Jacques, Here is a log file produced by running the command: mpirun -np 6 IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm -v > outfile.log I know that --use-hdf5-node-comm is supposed to use a lot of memory but it seems like it was using too much for my case. When producing this log file, by the time I stopped the simulation, it was using 30GB of RAM and still steadily increasing. I have had crashes on clusters before by exceeding the memory limit with similar cases to this. I remedied it those other times by coarsening the mesh which how I arrived at the current mesh. I can't afford to coarsen the mesh further. Perhaps this issue with the memory is a clue to finding the problem. Thank you, Isaac ________________________________ From: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Sent: Wednesday, August 9, 2023 1:34 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, I am not sure why your simulation is so slow but providing a log file could be helpful: srun IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm > outfile.log Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Sent: Wednesday, August 9, 2023 7:44 PM To: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>>; Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Jacques, In release mode. Regards, Isaac ________________________________ From: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Sent: Wednesday, August 9, 2023 12:41 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, Did you compile Nektar++ in release or debug mode? Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Sent: Wednesday, August 9, 2023 7:36 PM To: Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>>; Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi all, I forgot to remove the .opt file from the folder I sent. The issue is not with the collections. I have experimented with them and found that starting with no .opt file when running a simulation and letting Nektar++ create it automatically gives the best results. I always do this now, but the gains are not significant enough. Here is the drive folder again: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... It may be that this is not an optimisation issue but rather a debugging one. I feel that the issue is deeper than collections. I ran the same case on very similar meshes in OpenFOAM and it was also slower than it should have been. Would any of you be able to run this briefly on your own machines and tell me: 1) how long it took to save three or so time steps and 2) how many cores you ran on? If you are unable to do this, do you know of any other optimization/debugging techniques? Thank you very much, Isaac ________________________________ From: Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Sent: Wednesday, August 9, 2023 10:42 AM To: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: Re: [Nektar-users] Optimisation Issues [△EXTERNAL] Thank you Jacques Here is the .opt file generated: <NEKTAR> <COLLECTIONS> <OPERATOR TYPE="BwdTrans"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="Helmholtz"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="NoCollection" /> </OPERATOR> <OPERATOR TYPE="IProductWRTBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="IProductWRTDerivBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="PhysDeriv"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="MatrixFree" /> </OPERATOR> </COLLECTIONS> </NEKTAR> I had manually added <COLLECTIONS DEFAULT="auto" /> to my session file. Are the above results of the latter? Kind regards syavash On Wed, Aug 9, 2023 at 6:13 PM Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> wrote: Hi Ehsan, The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”. Best, Jacques From: nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk> <nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk>> On Behalf Of Ehsan Asgari Sent: Wednesday, August 9, 2023 3:23 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Isaac I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run. Could you let me know if it had a significant influence? Kind regards syavash On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> wrote: This email from isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
participants (3)
-
Ehsan Asgari
-
Isaac Rosin
-
Jacques Xing