September 2023 - Nektar-users

Using FieldConvert for parallel solution files
by Nande, Ishan A. 30 Sep '23

30 Sep '23

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hello, I hope everyone is ok. I am a new user of Nektar++ and in a bind. Specifically, I am stuck on using the FieldConvert utility to re-construct the decomposed solution files generated when the AcousticSolver was run with mpirun.I have referred to the user guide Section 5.7(Version 5.3.0) but it doesn't work. I tried to run the FieldConvert with 10 processors like this: mpirun -np 10 FieldConvert case.xml case_0.chk case_0.vtu. This didn't work and throws errors. This syntax matches with what is mentioned in Section 5.7 of the user guide. What would be the right way to using FieldConvert when you have solution files that have been generated in parallel run ? Regards, Ishan PGR

2 2

TurbChFl_3DH1D.xml for IncNaverStokesSolver
by Jialin Su 29 Sep '23

29 Sep '23

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Dear colleagues, I am try to run the turbulent channel flow example from the user guide. However, I am not able to find 'TurbChFl_3DH1D.xml' anywhere. Could you please let me advise on where I can find this setup file? The same applies to 'Pipe_turb.xml'. Thank you. Kind regards, Dr Jialin Su Lecturer in Fluid Dynamics Web: https://www.lboro.ac.uk/departments/aae/staff/jialin-su Department of Aeronautical and Automotive Engineering Loughborough University, Loughbororough, Leicestershire, LE11 3TU, UK Tel: +44 (0)1509 227204

1 0

Website Down?
by Isaac Rosin 28 Sep '23

28 Sep '23

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Hello Nektar Community, Is part of the Nektar website currently down? I have been unable to install the third party libraries while setting up Nektar on a cluster which wasn't an issue before. I can access the front page of the Nektar website, but when I click the links for the user guide, I get error 404 for the html version and error 403 for the PDF. The same thing happens with the tutorials. Kind regards, Isaac

5 5

Macro-element footprint on ReynoldsStress
by Ehsan Asgari 28 Sep '23

28 Sep '23

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Dear All I am doing a quasi 3-d simulation with a periodic direction in Z. When I use the "ReynoldsStresses" filter, I get a smooth time-averaged velocity, but I see the footprints of macro-elements on the Reynolds stress contours. I wonder why this can happen, as the Reynolds stresses are derived directly from the velocity and as far as I can tell, the representation of velocity contours is fine. Is there any reason that can explain this? Kind regards syavash

2 2

Provide symmetric initial random noise with respect to channel centerline
by Nandan Sarkar 21 Sep '23

21 Sep '23

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Dear Nektar++ users, I am performing Floquet stability analysis in a 2D channel. I am using the 'awgn' function for the initial random seed values for velocities in the channel likewise: <FUNCTION NAME="InitialConditions"> <E VAR="u" VALUE="awgn(0.01)" /> <E VAR="v" VALUE="awgn(0.01)" /> <E VAR="p" VALUE="0" /> </FUNCTION> I require the Initial Conditions (as shown above) to be symmetric with respect to channel centerline. Is there a way to achieve the same? Thanks and Regards Nandan Sarkar Research Scholar Department of Mechanical Engineering Indian Institute of Technology Kharagpur

1 0

Outlet flow causing simulations to crash
by Matt Duran 06 Sep '23

06 Sep '23

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Dear Nektar++ users, I have been experiencing issues related to flow in the outlet region of my simulations. I have been running flow past a cylinder at Reynolds number Re = 3900, on some occasions when the vortexes being shed reach the outlet, they cause my simulation to crash. This problem persists for 2D and 3D simulations. I have tried several fixes such as a finer time step of 5E-5 and making the mesh very high resolution. I have also tried increasing the distance from the cylinder to the outlet, with the outlet being located 40 cylinder diameters downstream from the cylinder. And I have also tried using a higher resolution 'buffer zone' near the outlet and also making the mesh unstructured near the outlet but I still experience crashing. The issue seems to be that the vortexes get stuck in the outlet boundary and then begin sending pressure and velocity backwards upstream into the flow causing the simulation to crash. I would greatly appreciate it if anyone had a potential solution or work around to this. I can provide my simulation files if it would help. Thanks, Matt Duran

2 2

Re: [Nektar-users] Optimisation Issues
by Isaac Rosin 06 Sep '23

06 Sep '23

Hi all, Thank you for your advice. I know it has been a while but I would still like to reply to this thread with some new findings which may help users in the future. Jacques, I tried different preconditioners, playing with tolerances, and other global system solver settings. I was able to speed up computations, but not by as much as I had hoped. I did reduce the memory consumption enough through this though. I had a look at some of the Nek example cases for a similar geometry. From this, I learned that by far the best way to reduce memory consumption and CPU time is by coarsening the mesh. I also found that by doing this, my model was no less capable of resolving the mechanisms in the flow I had intended. In fact, I found I was able to resolve them better than with the finer mesh. The capabilities of this solver even in large and high aspect ratio mesh cells are very impressive. Kind regards, Isaac ________________________________ From: Jacques Xing <jacques.xing(a)kcl.ac.uk> Sent: Thursday, August 10, 2023 4:04 AM To: Isaac Rosin <isaac.rosin1(a)ucalgary.ca> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, I am not sure of the exact cause of your large memory consumption, but I am aware of the large memory footprint of hdf5. Best, Jacques From: Isaac Rosin <isaac.rosin1(a)ucalgary.ca> Sent: Wednesday, August 9, 2023 10:01 PM To: Jacques Xing <jacques.xing(a)kcl.ac.uk>; Ehsan Asgari <eh.asgari(a)gmail.com> Cc: Maziyar Hassanpour <maziyar.hassanpour(a)ucalgary.ca>; nektar-users(a)imperial.ac.uk Subject: Re: [Nektar-users] Optimisation Issues Hi Jacques, I will investigate the solver info. Do you have any ideas about why the simulation may be using so much memory? Could it have anything to do with my mesh? Before I wasn't selecting the Optimize 3D option in gmsh. After trying it, I haven't noticed any improvements, but perhaps there are other areas of the mesh preparation I could look at? Thank you for your help so far. Best, Isaac ________________________________ From: Jacques Xing <jacques.xing(a)kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Sent: Wednesday, August 9, 2023 2:51 PM To: Isaac Rosin <isaac.rosin1(a)ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Ehsan Asgari <eh.asgari(a)gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour(a)ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, Memory issues might be a problem. However, I took a quick look at the log file and I would also point out that the number of CG (conjugate gradient) iterations can be quite large. Some improvement might be achieved by changing the parameters of the linear system solvers, specified within the following tag <GLOBALSYSSOLNINFO>. I refer you to the documentation for more details. Best, Jacques From: Isaac Rosin <isaac.rosin1(a)ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Sent: Wednesday, August 9, 2023 9:26 PM To: Jacques Xing <jacques.xing(a)kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>>; Ehsan Asgari <eh.asgari(a)gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour(a)ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Jacques, Here is a log file produced by running the command: mpirun -np 6 IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm -v > outfile.log I know that --use-hdf5-node-comm is supposed to use a lot of memory but it seems like it was using too much for my case. When producing this log file, by the time I stopped the simulation, it was using 30GB of RAM and still steadily increasing. I have had crashes on clusters before by exceeding the memory limit with similar cases to this. I remedied it those other times by coarsening the mesh which how I arrived at the current mesh. I can't afford to coarsen the mesh further. Perhaps this issue with the memory is a clue to finding the problem. Thank you, Isaac ________________________________ From: Jacques Xing <jacques.xing(a)kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Sent: Wednesday, August 9, 2023 1:34 PM To: Isaac Rosin <isaac.rosin1(a)ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Ehsan Asgari <eh.asgari(a)gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour(a)ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, I am not sure why your simulation is so slow but providing a log file could be helpful: srun IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm > outfile.log Best, Jacques From: Isaac Rosin <isaac.rosin1(a)ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Sent: Wednesday, August 9, 2023 7:44 PM To: Jacques Xing <jacques.xing(a)kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>>; Ehsan Asgari <eh.asgari(a)gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour(a)ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Jacques, In release mode. Regards, Isaac ________________________________ From: Jacques Xing <jacques.xing(a)kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Sent: Wednesday, August 9, 2023 12:41 PM To: Isaac Rosin <isaac.rosin1(a)ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Ehsan Asgari <eh.asgari(a)gmail.com<mailto:eh.asgari@gmail.com>> Cc: Maziyar Hassanpour <maziyar.hassanpour(a)ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: RE: [Nektar-users] Optimisation Issues [△EXTERNAL] Hi Isaac, Did you compile Nektar++ in release or debug mode? Best, Jacques From: Isaac Rosin <isaac.rosin1(a)ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Sent: Wednesday, August 9, 2023 7:36 PM To: Ehsan Asgari <eh.asgari(a)gmail.com<mailto:eh.asgari@gmail.com>>; Jacques Xing <jacques.xing(a)kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Maziyar Hassanpour <maziyar.hassanpour(a)ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi all, I forgot to remove the .opt file from the folder I sent. The issue is not with the collections. I have experimented with them and found that starting with no .opt file when running a simulation and letting Nektar++ create it automatically gives the best results. I always do this now, but the gains are not significant enough. Here is the drive folder again: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?us… It may be that this is not an optimisation issue but rather a debugging one. I feel that the issue is deeper than collections. I ran the same case on very similar meshes in OpenFOAM and it was also slower than it should have been. Would any of you be able to run this briefly on your own machines and tell me: 1) how long it took to save three or so time steps and 2) how many cores you ran on? If you are unable to do this, do you know of any other optimization/debugging techniques? Thank you very much, Isaac ________________________________ From: Ehsan Asgari <eh.asgari(a)gmail.com<mailto:eh.asgari@gmail.com>> Sent: Wednesday, August 9, 2023 10:42 AM To: Jacques Xing <jacques.xing(a)kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> Cc: Isaac Rosin <isaac.rosin1(a)ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>>; Maziyar Hassanpour <maziyar.hassanpour(a)ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> <nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk>> Subject: Re: [Nektar-users] Optimisation Issues [△EXTERNAL] Thank you Jacques Here is the .opt file generated: <NEKTAR> <COLLECTIONS> <OPERATOR TYPE="BwdTrans"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="Helmholtz"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="NoCollection" /> </OPERATOR> <OPERATOR TYPE="IProductWRTBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="IProductWRTDerivBase"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" /> </OPERATOR> <OPERATOR TYPE="PhysDeriv"> <ELEMENT TYPE="H" ORDER="*" IMPTYPE="MatrixFree" /> </OPERATOR> </COLLECTIONS> </NEKTAR> I had manually added <COLLECTIONS DEFAULT="auto" /> to my session file. Are the above results of the latter? Kind regards syavash On Wed, Aug 9, 2023 at 6:13 PM Jacques Xing <jacques.xing(a)kcl.ac.uk<mailto:jacques.xing@kcl.ac.uk>> wrote: Hi Ehsan, The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”. Best, Jacques From: nektar-users-bounces(a)imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk> <nektar-users-bounces(a)imperial.ac.uk<mailto:nektar-users-bounces@imperial.ac.uk>> On Behalf Of Ehsan Asgari Sent: Wednesday, August 9, 2023 3:23 PM To: Isaac Rosin <isaac.rosin1(a)ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> Cc: Maziyar Hassanpour <maziyar.hassanpour(a)ucalgary.ca<mailto:maziyar.hassanpour@ucalgary.ca>>; nektar-users(a)imperial.ac.uk<mailto:nektar-users@imperial.ac.uk> Subject: Re: [Nektar-users] Optimisation Issues Hi Isaac I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run. Could you let me know if it had a significant influence? Kind regards syavash On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1(a)ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca>> wrote: This email from isaac.rosin1(a)ucalgary.ca<mailto:isaac.rosin1@ucalgary.ca> originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list<https://spam.ic.ac.uk/SpamConsole/Senders.aspx> to disable email stamping for this address. Hello Nektar Community, I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?us… the slurm file called P4.sh has the script I normally use on clusters. I look forward to hearing from you! All the best, Isaac _______________________________________________ Nektar-users mailing list Nektar-users(a)imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users

1 0

average planes in a quasi 3-D simulation
by Ehsan Asgari 02 Sep '23

02 Sep '23

******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Dear All, How can I average the solutions of planes across the homogenous direction in a Quasi 3-D simulation? Kind regards syavash

2 5