Hi Isaac, Did you compile Nektar++ in release or debug mode? Best, Jacques From: Isaac Rosin <isaac.rosin1@ucalgary.ca> Sent: Wednesday, August 9, 2023 7:36 PM To: Ehsan Asgari <eh.asgari@gmail.com>; Jacques Xing <jacques.xing@kcl.ac.uk> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk Subject: Re: [Nektar-users] Optimisation Issues Hi all, I forgot to remove the .opt file from the folder I sent. The issue is not with the collections. I have experimented with them and found that starting with no .opt file when running a simulation and letting Nektar++ create it automatically gives the best results. I always do this now, but the gains are not significant enough. Here is the drive folder again: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... It may be that this is not an optimisation issue but rather a debugging one. I feel that the issue is deeper than collections. I ran the same case on very similar meshes in OpenFOAM and it was also slower than it should have been. Would any of you be able to run this briefly on your own machines and tell me: 1) how long it took to save three or so time steps and 2) how many cores you ran on? If you are unable to do this, do you know of any other optimization/debugging techniques? Thank you very much, Isaac ________________________________ From: Ehsan Asgari <eh.asgari@gmail.com<mailto:eh.asgari@gmail.com>> Sent: Wednesday, August 9, 2023 10:42 AM To: Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: Re: [Nektar-users] Optimisation Issues [△EXTERNAL] Thank you Jacques Here is the .opt file generated: <NEKTAR> <COLLECTIONS> <OPERATOR TYPE="BwdTrans"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="Helmholtz"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="NoCollection" /> </OPERATOR> <OPERATOR TYPE="IProductWRTBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="IProductWRTDerivBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="PhysDeriv"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="MatrixFree" /> </OPERATOR> </COLLECTIONS> </NEKTAR> I had manually added <COLLECTIONS DEFAULT="auto" /> to my session file. Are the above results of the latter? Kind regards syavash On Wed, Aug 9, 2023 at 6:13 PM Jacques Xing <jacques.xing@kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> wrote: Hi Ehsan, The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”. Best, Jacques From: nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk> <nektar-users-bounces@imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk>> On Behalf Of Ehsan Asgari Sent: Wednesday, August 9, 2023 3:23 PM To: Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users@imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Isaac I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run. Could you let me know if it had a significant influence? Kind regards syavash On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> wrote: This email from isaac.rosin1@ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp... the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac _______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users