Hi Vishal,
Dave Moxey emailed me earlier to say he had tried it too and there are some rather obvious memory mismanagement we need to sort out for the Hex meshes. Whilst we do this have you tried running this case with the 2.5D solver as shown in the tutorial. This has similar performance to the 2D solver.
I will also have a look at Dave suggestions and get back to you.
CheersSpencer
Sent from my iPhoneVishalRegards,Command run:Dear Spencer,Thank you for your reply. Yes, I have compiled Nektar++ with MPI (Although I get a warning on the cluster: WARN Conflicting CPU frequencies detected, using: 2900.06. The admin claims it's harmless and they're working on it).
For reference, I'm attaching the output log in compact and verbose mode, where the code seems to partition the mesh (??).
"""
JobID: 124250
======
Time: Thu 15 Mar 12:09:00 GMT 2018
Running on master node: node0367
Current directory: /gpfs/home/lboro/ttvs3/work_Saini/cases_Nektar/tgv/Re1600_ LES/N5/256cube
numtasks=112, numnodes=4, mpi_tasks_per_node=28 (OMP_NUM_THREADS=1)
Executing command:
==================
time mpirun -npernode 28 -np 112 IncNavierStokesSolver -v 2pi_domain_64cubeEle_mesh_copyPaste.xml tgv_conditions_lowTolerance. xml > log_124250
"""
It is a bit unlikely that there are 112 instances running in parallel because it is 256^3~17M degrees of freedom case and each instance most probably is bigger than 450GB/112~4GB.Hope these are helpful.
On Sun, Mar 25, 2018 at 7:09 PM, Sherwin, Spencer J <s.sherwin@imperial.ac.uk> wrote:
Dear Vishal,
Sorry for not getting to your comments earlier. Can I first confirm you have indeed compiled the code in parallel since what you describe sounds as if the code may have been compiled in serial and is running 112 versions of the case?
To turn on parallelisation you need to enable the NEKTAR_USE_MPI in the cmake step.
Best regards,Spencer.
On 19 Mar 2018, at 17:21, Vishal Saini <vishal.saini.nitj@gmail.com> wrote:
<nektar_user_tgv.tar.gz>______VishalBest regards,http://www.hpc-midlands-plus.aHi all,After obtaining exciting error vs. computational cost results from relatively heavy 2D computations, I've been setting up the 3D Taylor Green vortex (TGV) problem using Nektar++ Incompressible solver. The Reynolds number used is 1600 (same as in the TGV tutorial from Nektar++ team). A 3D grid of 64^3 elements with NumModes=5 was used to obtain a 256^3 simulation (close to a DNS). I ran this problem on Midlands+ Tier 2 machine using 4 nodes (112 cores, 512 GB RAM in total).
c.uk/about/system-description/
The memory consumption during the run was ~450GB and it took ~330 wall-clock-minutes for 1000 time-steps (with dt=1e-4). For a reference, I ran a 256^3 TGV simulation of the same case in OpenFOAM using same resources. The memory consumption was ~40GB and it took ~20 minutes for 1000 time-steps.
So for this case, I observe high resource consumption in terms of memory and computation time. I was not expecting this on the basis of my experience with Nektar++ on 2D simulations, at least in terms of computation time.
Is this normal? In addition to my set-up file, I'm suspecting my installation of Nektar++ on the aforementioned cluster. Could anyone please try and run a few hundred time-steps on their machines using the files attached? FYI, I'm partitioning on-the-fly. The calculation seems to scale well on 8 nodes in terms of wall clock time while increasing the memory consumption by ~10%.
Any input is highly appreciated.
---Master of Research,
Vishal SAINI
University of Cambridge.
______________________________ ___________
Nektar-users mailing list
Nektar-users@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
Spencer Sherwin FREng, FRAeS
Head, Aerodynamics,Professor of Computational Fluid Mechanics,Department of Aeronautics,
Imperial College LondonSouth Kensington Campus,http://www.imperial.ac.uk/people/s.sherwin/
<log.dat><log_verbose.dat>