Hi Jacques,

I will investigate the solver info.

Do you have any ideas about why the simulation may be using so much memory? Could it have anything to do with my mesh? Before I wasn't selecting the Optimize 3D option in gmsh. After trying it, I haven't noticed any improvements, but perhaps there are other areas of the mesh preparation I could look at?

Thank you for your help so far.

Best,
Isaac
From: Jacques Xing <jacques.xing@kcl.ac.uk>
Sent: Wednesday, August 9, 2023 2:51 PM
To: Isaac Rosin <isaac.rosin1@ucalgary.ca>; Ehsan Asgari <eh.asgari@gmail.com>
Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk <nektar-users@imperial.ac.uk>
Subject: RE: [Nektar-users] Optimisation Issues
 
[△EXTERNAL]


Hi Isaac,

 

Memory issues might be a problem. However, I took a quick look at the log file and I would also point out that the number of CG (conjugate gradient) iterations can be quite large. Some improvement might be achieved by changing the parameters of the linear system solvers, specified within the following tag <GLOBALSYSSOLNINFO>. I refer you to the documentation for more details.

 

Best,

Jacques

 

From: Isaac Rosin <isaac.rosin1@ucalgary.ca>
Sent: Wednesday, August 9, 2023 9:26 PM
To: Jacques Xing <jacques.xing@kcl.ac.uk>; Ehsan Asgari <eh.asgari@gmail.com>
Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk
Subject: Re: [Nektar-users] Optimisation Issues

 

Hi Jacques,

 

Here is a log file produced by running the command:

 

mpirun -np 6 IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm -v > outfile.log

 

I know that --use-hdf5-node-comm is supposed to use a lot of memory but it seems like it was using too much for my case. When producing this log file, by the time I stopped the simulation, it was using 30GB of RAM and still steadily increasing. I have had crashes on clusters before by exceeding the memory limit with similar cases to this. I remedied it those other times by coarsening the mesh which how I arrived at the current mesh. I can't afford to coarsen the mesh further. Perhaps this issue with the memory is a clue to finding the problem.

 

Thank you,

Isaac

From: Jacques Xing <jacques.xing@kcl.ac.uk>
Sent: Wednesday, August 9, 2023 1:34 PM
To: Isaac Rosin <isaac.rosin1@ucalgary.ca>; Ehsan Asgari <eh.asgari@gmail.com>
Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk <nektar-users@imperial.ac.uk>
Subject: RE: [Nektar-users] Optimisation Issues

 

[△EXTERNAL]

 

Hi Isaac,

 

I am not sure why your simulation is so slow but providing a log file could be helpful:

 

srun IncNavierStokesSolver session.xml --io-format Hdf5 --use-hdf5-node-comm > outfile.log

 

 

Best,

Jacques

 

 

From: Isaac Rosin <isaac.rosin1@ucalgary.ca>
Sent: Wednesday, August 9, 2023 7:44 PM
To: Jacques Xing <jacques.xing@kcl.ac.uk>; Ehsan Asgari <eh.asgari@gmail.com>
Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk
Subject: Re: [Nektar-users] Optimisation Issues

 

Hi Jacques,

 

In release mode.

 

Regards,

Isaac

From: Jacques Xing <jacques.xing@kcl.ac.uk>
Sent: Wednesday, August 9, 2023 12:41 PM
To: Isaac Rosin <isaac.rosin1@ucalgary.ca>; Ehsan Asgari <eh.asgari@gmail.com>
Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk <nektar-users@imperial.ac.uk>
Subject: RE: [Nektar-users] Optimisation Issues

 

[△EXTERNAL]

 

Hi Isaac,

 

Did you compile Nektar++ in release or debug mode?

 

Best,

Jacques

 

 

From: Isaac Rosin <isaac.rosin1@ucalgary.ca>
Sent: Wednesday, August 9, 2023 7:36 PM
To: Ehsan Asgari <eh.asgari@gmail.com>; Jacques Xing <jacques.xing@kcl.ac.uk>
Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk
Subject: Re: [Nektar-users] Optimisation Issues

 

Hi all,

 

I forgot to remove the .opt file from the folder I sent. The issue is not with the collections. I have experimented with them and found that starting with no .opt file when running a simulation and letting Nektar++ create it automatically gives the best results. I always do this now, but the gains are not significant enough. Here is the drive folder again: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp=sharing

 

It may be that this is not an optimisation issue but rather a debugging one. I feel that the issue is deeper than collections. I ran the same case on very similar meshes in OpenFOAM and it was also slower than it should have been. Would any of you be able to run this briefly on your own machines and tell me: 1) how long it took to save three or so time steps and 2) how many cores you ran on? If you are unable to do this, do you know of any other optimization/debugging techniques?

 

Thank you very much,

Isaac

From: Ehsan Asgari <eh.asgari@gmail.com>
Sent: Wednesday, August 9, 2023 10:42 AM
To: Jacques Xing <jacques.xing@kcl.ac.uk>
Cc: Isaac Rosin <isaac.rosin1@ucalgary.ca>; Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk <nektar-users@imperial.ac.uk>
Subject: Re: [Nektar-users] Optimisation Issues

 

[△EXTERNAL]

 

Thank you Jacques

 

Here is the .opt file generated:

 

<NEKTAR>
    <COLLECTIONS>
        <OPERATOR TYPE="BwdTrans">
            <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" />
        </OPERATOR>
        <OPERATOR TYPE="Helmholtz">
            <ELEMENT TYPE="H" ORDER="*" IMPTYPE="NoCollection" />
        </OPERATOR>
        <OPERATOR TYPE="IProductWRTBase">
            <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" />
        </OPERATOR>
        <OPERATOR TYPE="IProductWRTDerivBase">
            <ELEMENT TYPE="H" ORDER="*" IMPTYPE="SumFac" />
        </OPERATOR>
        <OPERATOR TYPE="PhysDeriv">
            <ELEMENT TYPE="H" ORDER="*" IMPTYPE="MatrixFree" />
        </OPERATOR>
    </COLLECTIONS>
</NEKTAR>

 

I had manually added  <COLLECTIONS DEFAULT="auto" /> to my session file. Are the above results of the latter?

 

Kind regards

syavash

 

 

 

 

On Wed, Aug 9, 2023 at 6:13 PM Jacques Xing <jacques.xing@kcl.ac.uk> wrote:

Hi Ehsan,

 

The optimization is likely to be problem and architecture dependent. Normally, Nektar++ automatically produce an optimization file *.opt. You should be able to see it. If you want to examine the influence of optimization on computational time, I guess you can manually set IMPTYPE to “NoCollection”.

 

Best,

Jacques

 

From: nektar-users-bounces@imperial.ac.uk <nektar-users-bounces@imperial.ac.uk> On Behalf Of Ehsan Asgari
Sent: Wednesday, August 9, 2023 3:23 PM
To: Isaac Rosin <isaac.rosin1@ucalgary.ca>
Cc: Maziyar Hassanpour <maziyar.hassanpour@ucalgary.ca>; nektar-users@imperial.ac.uk
Subject: Re: [Nektar-users] Optimisation Issues

 

Hi Isaac

 

I am doing a simulation on a similar configuration, a wall-mounted cylinder with Re=750. It's actually the first time I read about the optimization technique to speed up the run.

Could you let me know if it had a significant influence?

 

Kind regards

syavash

 

 

On Wed, Aug 9, 2023 at 4:38 AM Isaac Rosin <isaac.rosin1@ucalgary.ca> wrote:

This email from isaac.rosin1@ucalgary.ca originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list to disable email stamping for this address.

 

Hello Nektar Community,

 

I am running a full 3D DNS simulation at Re=400 of a flow past a rectangular prism using the incompressible solver. This flow is laminar but 3D effects are present. I am using a structured mesh generated using gmsh with a p order of 4 which comes to roughly 2.6 million degrees of freedom. A colleague is able to run a similar but slightly more complex DNS simulation at Re=3900 on 1/8 the number of processors and he still obtains results much faster. All signs indicate I should be able to use far fewer resources than I currently am, but neither of us can figure out how to optimise further than I already have. I followed the optimisation section of the Nektar user guide, tweaked my mesh several times, changed the time step size, etc., but all of these adjustments showed marginal improvements at best. I would be very grateful if anyone has any suggestions. I am providing my case files here: https://drive.google.com/drive/folders/1iT0GvZgE0MyfmjNe1GYMlAKJp-8fh57X?usp=sharing the slurm file called P4.sh has the script I normally use on clusters.

 

I look forward to hearing from you!

 

All the best,

Isaac

_______________________________________________
Nektar-users mailing list
Nektar-users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/nektar-users