Hi all,
After obtaining exciting error vs. computational cost results from relatively heavy 2D computations, I've been setting up the 3D Taylor Green vortex (TGV) problem using Nektar++ Incompressible solver. The Reynolds number used is 1600 (same as in the TGV tutorial
from Nektar++ team). A 3D grid of 64^3 elements with NumModes=5 was used to obtain a 256^3 simulation (close to a DNS). I ran this problem on Midlands+ Tier 2 machine using 4 nodes (112 cores, 512 GB RAM in total).
http://www.hpc-midlands-plus.ac.uk/about/system-description/
The memory consumption during the run was ~450GB and it took ~330 wall-clock-minutes for 1000 time-steps (with dt=1e-4). For a reference, I ran a 256^3 TGV simulation of the same case in OpenFOAM using same resources. The memory consumption was ~40GB and it
took ~20 minutes for 1000 time-steps.
So for this case, I observe high resource consumption in terms of memory and computation time. I was not expecting this on the basis of my experience with Nektar++ on 2D simulations, at least in terms of computation time.
Is this normal? In addition to my set-up file, I'm suspecting my installation of Nektar++ on the aforementioned cluster. Could anyone please try and run a few hundred time-steps on their machines using the files attached? FYI, I'm partitioning on-the-fly. The
calculation seems to scale well on 8 nodes in terms of wall clock time while increasing the memory consumption by ~10%.
Any input is highly appreciated.