May 2016 - Nektar-users - imperial.ac.uk

Calculating memory for a job
by Amitvikram Dutta 05 Jun '16

05 Jun '16

Hi everyone, How would I go about calculating the memory requirements necessary for a particular job? I have a 23k cell mesh problem, that keeps getting deleted from my cluster because of exceeding memory usage requirements. Sincerely, -- *Amitvikram Dutta* *MASc Candidate* *Graduate Research Assistant * *Okanagan CFD Laboratory* *University of British Columbia | Okanagan Campus*

2 5

Re: [Nektar-users] Parallel transposition not implemented yet for 3D-Homo-2D approach
by Asim Onder 01 Jun '16

01 Jun '16

Hi Douglas, thanks for the feedback. I was aware of --npz parallelization but was using a small number, not 1/2 or 1/4 of HomModesZ. Increasing npz really helped. I still have to try GlobalSysSoln. Now I face a memory problem for another case. The simulation runs out of memory when starting from a checkpoint file. Here is a little bit information about this case: - Mesh is made of around 16000 quad elements with p=5, i.e., NUMMODES="6" TYPE="MODIFIED" in xy, and HomModesZ=1080 in z direction. - I'm trying to run this case on 60 computing nodes each equipped with 24 processors, and a memory of 105 gb. In total, it makes 1440 procs, and 6300gb memory. - Execution command: mpirun -np 1440 IncNavierStokesSolver --npz 360 config.xml I was wondering if the memory usage of the application is scaling on different cores during IO, or using only one core. If it is only one core, than if it exceeds 105gb, it crushes I guess. Would you have maybe any suggestion/comment on this? Thanks, Asim On 04/13/2016 12:12 AM, Serson, Douglas wrote: Hi Asim, Concerning your questions: 1- Are you using the command line argument --npz? This is very important for obtaining an efficient parallel performance with the Fourier expansion, since it defines the number of partitions in the z-direction. If it is not set, only the xy plane will be partitioned and the parallelism will saturate quickly. I suggest initially setting npz to 1/2 or 1/4 of HomModesZ (note that nprocs must be a multiple of npz, since nprocs/npz is the number of partitions in the xy plane). Also, depending on your particular case and the number of partitions you have in the xy plane, your simulation may benefit from using a direct solver for the linear systems. This can be activated by adding '-I GlobalSysSoln=XxtMultiLevelStaticCond' to the command line. This is usually more efficient for a small number of partitions, but considering the large size of your problem it might be worth trying it. 2- I am not sure what could be causing that. I suppose it would help if you could send the exact commands you are using to run FieldConvert. Cheers, Douglas ________________________________ From: Asim Onder <ceeao(a)nus.edu.sg><mailto:ceeao@nus.edu.sg> Sent: 12 April 2016 06:42 To: Sherwin, Spencer J; Serson, Douglas Cc: nektar-users; Moxey, David Subject: Re: [Nektar-users] Parallel transposition not implemented yet for 3D-Homo-2D approach Dear Spencer, Douglas, Nektar-users, I'm involved now in testing of a local petascale supercomputer, and for some quite limited time I can use several thousand processors for my DNS study. My test case is oscillating flow over a rippled bed. I build up a dense unstructured grid with p=6 quadrilateral elements in x-y, and Fourier expansions in z directions. In total I have circa half billion dofs per variable. I would have a few questions about this relatively large case: 1. I noticed that scaling gets inefficient after around 500 procs, let's say parallel efficiency goes below 80%. I was wondering if you would have any general suggestions to tune the configurations for a better scaling. 2. Postprocessing vorticity and Q criterion is not working for this case. At the of the execution Fieldconvert writes some small files without the field data. What could be the reason for this? Thanks you in advance for your suggestions. Cheers, Asim On 03/21/2016 04:16 AM, Sherwin, Spencer J wrote: Hi Asim, To follow-up on Douglas’ comment we are trying to get more organised to sort out a developers guide. We are also holding a user meeting in June. If you were able to make this we could also try and have a session on getting you going on the developmental side of things. Cheers, Spencer. On 17 Mar 2016, at 14:58, Serson, Douglas <d.serson14(a)imperial.ac.uk<mailto:d.serson14@imperial.ac.uk>> wrote: Hi Asim, I am glad that your simulation is now working. About your questions: 1. We have some work done on a filter for calculating Reynolds stresses as the simulation progresses, but it is not ready yet, and it would not provide all the statistics you want. Since you already have a lot of chk files, I suppose the best way would indeed be using a script to process all of them with FieldConvert. 2. Yes, this has been recently included in FieldConvert, using the new 'meanmode' module. 3. I just checked that, and apparently this is caused by a bug when using this module without fftw. This should be fixed soon, but as an alternative this module should work if you switch fftw on (just add <I PROPERTY="USEFFT" VALUE="FFTW"/> to you session file, if the code was compiled with support to fftw). 4. I think there is some work towards a developer guide, but I don't how advanced is the progress on that. I am sure Spencer will be able to provide you with more information on that. Cheers, Douglas ________________________________________ From: Asim Onder <ceeao(a)nus.edu.sg<mailto:ceeao@nus.edu.sg>> Sent: 17 March 2016 09:10 To: Serson, Douglas; Sherwin, Spencer J Cc: nektar-users; Moxey, David Subject: Re: [Nektar-users] Parallel transposition not implemented yet for 3D-Homo-2D approach Hi Spencer, Douglas, Thanks to your suggestions I managed to get the turbulent regime for the oscillatory channel flow. I have now completed the DNS study for one case, and built up a large database with checkpoint (*chk) files. I would like to calculate turbulent statistics using this database, especially for second order terms, e.g. Reynolds stresses and turbulent dissipation, and third order terms, e.g. turbulent diffusion terms. However, I am a little bit confused how I could achieve this. I would appreciate if you could give some hints about the following: 1. The only way I could think of to calculate turbulent statistics is to write a simple bash script to iterate over chk files, and apply various existing/extended FieldConvert operations on individual chk files. This would require some additional storage to store the intermediate steps, and therefore would be a bit cumbersome. Would it be any simpler way directly doing this directly in Nektar++? 2. I have one homogeneous direction, for which I used Fourier expansions. I would like to apply spatial averaging over this homogeneous direction. Does Nektar++ already contain such functionality? 3. I want to use 'wss' in Fieldconvert module to calculate wall shear stress. However, it returns segmentation fault. Any ideas why it could be? 4. I was wondering if there is any introductory document for basic programming in Nektar++. User guide does not contain information about programming. It would be nice to have some additional information to Doxygen documentation. Thank you very much in advance for your feedback. Cheers, Asim On 02/15/2016 11:59 PM, Serson, Douglas wrote: Hi Asim, As Spencer mentioned, svv can help in stabilizing your solution. You can find information on how to set it up in the user guide (pages 92-93), but basically all you need to do is use: <I PROPERTY="SpectralVanishingViscosity" VALUE="True"/> You can also tune it by setting the parameters SVVCutoffRatio and SVVDiffCoeff, but I would suggest starting with the default parameters. Also, you can use the parameter IO_CFLSteps to output the CFL number. This way you can check if the time step you are using is appropriate. Cheers, Douglas From: Sherwin, Spencer J Sent: 14 February 2016 19:46 To: ceeao Cc: nektar-users; Serson, Douglas; Moxey, David Subject: Re: [Nektar-users] Parallel transposition not implemented yet for 3D-Homo-2D approach Hi Asim, Getting a flow through transition is very challenging since there is a strong localisation of shear and this can lead to aliasing issues which can then cause instabilities. Both Douglas and Dave have experienced this with recent simulations so I am cc’ing them to make some suggestions. I would be inclined to be using spectralhpdealiasing and svv. Hopefully Douglas can send you an example of how to switch this on. Cheers, Spencer. On 11 Feb 2016, at 10:32, ceeao<<mailto:ceeao@nus.edu.sg>ceeao(a)nus.edu.sg<mailto:ceeao@nus.edu.sg>> wrote: Hi Spencer, Nektar-Users, I followed the suggestion and coarsened the grid a bit. This way it worked impressively fast, but the flow is stable and remains laminar, as I didn't add any perturbations. I need to kick the transition to have turbulence. If I add white noise, even very low magnitude, conjugate gradient solver blows up again. I also tried adding some sinusoidal perturbations to boundary conditions, and again had troubles with CG. I don't really get CG's extreme sensitivity to perturbations. Any suggestion is much appreciated. Thanks in advance. Cheers, Asim On 02/08/2016 04:48 PM, Sherwin, Spencer J wrote: HI Asim, How many parallel cores are you running on. Sometime starting up these flows can be tricky especially if you are immediately jumping to a high Reynolds number. Have you tried first starting the flow at a Lower Reynolds number? Also 100 x 200 is quite a few elements in the x-y plane. Remember the polynomial order adds in more points on top of the mesh discretisation. I would perhaps recommend trying a smaller mesh to see how that goes first. Actually I note there is a file called TurbChFl_3D1H.xml in the ~/Nektar/Solvers/IncNavierStokesSolver/Examples directory which might be worth looking at. I think this was a mesh used in Ale Bolis’ thesis which you can find under: http://wwwf.imperial.ac.uk/ssherw/spectralhp/papers/PhDThesis/Bolis_Thesis.… Cheers, Spencer. On 1 Feb 2016, at 07:01, ceeao<mailto:ceeao@nus.edu.sg><ceeao(a)nus.edu.sg><mailto:ceeao@nus.edu.sg> wrote: Hi Spencer, Thank you for the quick reply and suggestion. I switched indeed to 3D homo 1D case and this time I have problems with the divergence of linear solvers. I refined the grid in the channel flow example to 100x200x64 in x-y-z directions, and left everything else the same. When I employ the default global system solver "IterativeStaticCond" with this setup, I get divergence: "Exceeded maximum number of iterations (5000)". I checked the initial fields and mesh in Paraview, everything seems to be normal. I also tried the "LowEnergyBlock" preconditioner, and apparently this one is valid only in sheer 3D cases. My knowledge in iterative solvers for hp-Fem is minimal. Therefore, I was wondering if you could suggest maybe a robust option that at least converge. My concern is getting some rough estimates for the speed of Nektar++ in my oscillating channel flow problem. If the speed will be promising, I will switch to Nektar++ from OpenFOAM, as OpenFOAM is low-order and not really suitable for DNS. Thanks again in advance. Cheers, Asim On 01/31/2016 11:53 PM, Sherwin, Spencer J wrote: Hi Asim, I think your conclusions is correct. We did some early implementation into the 2D Homogeneous expansion but have not pulled it all the way through since we did not have a full project on this topic. We have however kept the existing code running through our regression test. For now I would perhaps suggest you try the 3D homo 1D approach for your runs since you can use parallelisation in that code. Cheers, Spencer. On 29 Jan 2016, at 04:00, ceeao<mailto:ceeao@nus.edu.sg><ceeao(a)nus.edu.sg><mailto:ceeao@nus.edu.sg> wrote: Dear all, I just installed the library, and need to simulate DNS of a channel flow with oscillating pressure gradient. As I have two homogeneous directions I applied standard Fourier discretization in these directions. It seems like this case is not parallelized yet, and I got the error in the subject. I was wondering if I'm overlooking something. If not, are there maybe any plans in the future to include parallelization of 2D FFT's? Thank you in advance. Best, Asim Onder Research Fellow National University of Singapore ________________________________ Important: This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify us immediately; you should not copy or use it for any purpose, nor disclose its contents to any other person. Thank you. _______________________________________________ Nektar-users mailing list Nektar-users(a)imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users Spencer Sherwin McLaren Racing/Royal Academy of Engineering Research Chair, Professor of Computational Fluid Mechanics, Department of Aeronautics, Imperial College London South Kensington Campus London SW7 2AZ s.sherwin(a)imperial.ac.uk<mailto:s.sherwin@imperial.ac.uk> +44 (0) 20 759 45052 Important: This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify us immediately; you should not copy or use it for any purpose, nor disclose its contents to any other person. Thank you. Spencer Sherwin McLaren Racing/Royal Academy of Engineering Research Chair, Professor of Computational Fluid Mechanics, Department of Aeronautics, Imperial College London South Kensington Campus London SW7 2AZ s.sherwin(a)imperial.ac.uk<mailto:s.sherwin@imperial.ac.uk> +44 (0) 20 759 45052 Important: This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify us immediately; you should not copy or use it for any purpose, nor disclose its contents to any other person. Thank you. Spencer Sherwin McLaren Racing/Royal Academy of Engineering Research Chair, Professor of Computational Fluid Mechanics, Department of Aeronautics, Imperial College London South Kensington Campus London SW7 2AZ s.sherwin(a)imperial.ac.uk<mailto:s.sherwin@imperial.ac.uk> +44 (0) 20 759 45052 ________________________________ Important: This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify us immediately; you should not copy or use it for any purpose, nor disclose its contents to any other person. Thank you. Spencer Sherwin McLaren Racing/Royal Academy of Engineering Research Chair, Professor of Computational Fluid Mechanics, Department of Aeronautics, Imperial College London South Kensington Campus London SW7 2AZ <mailto:s.sherwin@imperial.ac.uk>s.sherwin(a)imperial.ac.uk<mailto:s.sherwin@imperial.ac.uk> +44 (0) 20 759 45052 ________________________________ Important: This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify us immediately; you should not copy or use it for any purpose, nor disclose its contents to any other person. Thank you. ________________________________ Important: This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify us immediately; you should not copy or use it for any purpose, nor disclose its contents to any other person. Thank you.

3 18

Linking ARPAC
by Lander, Marcus David 27 May '16

27 May '16

I am having issues getting Nektar++ to recognize ARPACK. When I run through the channel-flow tutorial, in the Channel-Coupled.xml section, I get to the following command: IncNavierStokesSolver Channel-Coupled.xml Then, I receive the following error: Value 'Arpack' is not valid for property 'DRIVER' I'm very confused how to link Arpack (which is going to be installed to my personal directory rather than on the main section of the cluster, because I do not have permission to install Arpack globally). It is installed at $HOME/ARPACK/ In ccmake .., I point the ARPACK_LIBRARY flag to $HOME/ARPACK/libarpack_LINUX.a, and then I changed the NEKTAR_USE_ARPACK flag to on. Is there something I am missing???

2 1

Parallel Problem with FieldConvert
by Jian Yu 27 May '16

27 May '16

Dear all, I encountered a problem when I run the FieldConvert with two or more cpu cores for the C0Projection module. For example, when I run the following command, it is OK: mpirun -np 1 ./FieldConvert -m C0Projection 2D_20x20.xml 2D_20x20_0.chk 2D_20x20_0_C0.fld However, the following command will report an error: mpirun -np 2 ./FieldConvert -m C0Projection 2D_20x20.xml 2D_20x20_0.chk 2D_20x20_0_C0.fld and the error message is: gs_setup: 60 unique labels shared pairwise times (avg, min, max): 0.000110245 0.00010879 0.000111699 crystal router : 9.22918e-05 9.22918e-05 9.22918e-05 all reduce : 0.000179148 0.0001791 0.000179195 used all_to_all method: crystal router handle bytes (avg, min, max): 4724 4724 4724 buffer bytes (avg, min, max): 1920 1920 1920 CG iterations made = 5001 using tolerance of 1e-09 (error = nan, rhs_mag = inf) Fatal : Level 0 assertion violation Exceeded maximum number of iterations terminate called after throwing an instance of 'ErrorUtil::NekError' what(): Level 0 assertion violation Exceeded maximum number of iterations Can anyone help me about this? Best Jian

2 2

Modified Cylinder Example: Time Evolution of a Solution
by Lander, Marcus David 23 May '16

23 May '16

I am trying to figure out how to generate a time based output showing snapshots of the evolution of the flow field around a cylinder. In essence, I want Nektar++ to output the values of the entire flow field after each time step. Having read through the User-Guide, I believe that I am able to do this by using <FILTERS> <FILTER TYPE="Checkpoint"> <PARAM NAME="OutputFile">IntermediateFields</PARAM> <PARAM NAME="OutputFrequency">1</PARAM> </FILTER> </FILTERS> Which will output a .chk file after each time step. Based on my OutputFrequency flag, I should get a .chk file after every time step. I am running this using the files from the Cylinder Flow example, and I am modifying Cylinder-Base.xml. However, when I run this, I get a single "Cylinder-Base_0.chk" file as opposed to 10 files. Any support would be greatly appreciated. Marcus Lander Graduate Research Student Oklahoma State University Marcus.lander(a)okstate.edu

2 1

Re: [Nektar-users] Linking ARPAC
by Sherwin, Spencer J 23 May '16

23 May '16

Hi Marcus, Glad it all worked out in the end. I am posting your summary back onto the nectar-users list so we can keep a record. Cheers, Spencer. On 23 May 2016, at 21:30, Lander, Marcus David <marcus.lander(a)okstate.edu<mailto:marcus.lander@okstate.edu>> wrote: Dr. Spencer, I was able to successfully link ARPACK to NEKTAR++. I followed what you asked, and I compiled Nektar++ using the same compiler as I was using for Nektar++. However, this still lead to some errors. Additionally, I needed to change multiple compile flags within ARPACK’s various makefiles. Finally, I figured out that instead of linking to libarpack_LINUX.a, I instead needed to link to libarpack_LINUX.so. At this point, Nektar++ compiled correctly, and I was able to confirm it by calling Arpack as is described in the tutorials. Thank you very much. Marcus Lander Graduate Research Student Oklahoma State University CC’d in this email: Balaji Jayaraman, Ph.D, Assistant Professor, School of Mechanical and Aerospace Engineering, Oklahoma State University, 211C Engineering North, Stillwater OK 74078-5016 Email: balaji.jayaraman(a)okstate.edu<mailto:balaji.jayaraman@okstate.edu> From: Sherwin, Spencer J [mailto:s.sherwin@imperial.ac.uk] Sent: Friday, May 20, 2016 9:27 AM To: Lander, Marcus David Subject: Re: [Nektar-users] Linking ARPAC Hi Marcus What you seem to be doing seems reasonable. However when you enter the details in cmake you mention below. What happens when you then configure the setup by pressing ’c’. Does the configuration complete correctly? I am using it from a system installation which is actually linking to a dynamic library libarpack.dylib but I think you should be able to use the static compilation. Are you compelling Arpack with the same compiler you are using for Nektar++ (@Chris, Dave: Do you have any suggestions)? Cheers, Spencer. On 19 May 2016, at 22:11, Lander, Marcus David <marcus.lander(a)okstate.edu<mailto:marcus.lander@okstate.edu>> wrote: I am having issues getting Nektar++ to recognize ARPACK. When I run through the channel-flow tutorial, in the Channel-Coupled.xml section, I get to the following command: IncNavierStokesSolver Channel-Coupled.xml Then, I receive the following error: Value ‘Arpack’ is not valid for property ‘DRIVER’ I’m very confused how to link Arpack (which is going to be installed to my personal directory rather than on the main section of the cluster, because I do not have permission to install Arpack globally). It is installed at $HOME/ARPACK/ In ccmake .., I point the ARPACK_LIBRARY flag to $HOME/ARPACK/libarpack_LINUX.a, and then I changed the NEKTAR_USE_ARPACK flag to on. Is there something I am missing??? _______________________________________________ Nektar-users mailing list Nektar-users(a)imperial.ac.uk<mailto:Nektar-users@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users Spencer Sherwin McLaren Racing/Royal Academy of Engineering Research Chair, Professor of Computational Fluid Mechanics, Department of Aeronautics, Imperial College London South Kensington Campus London SW7 2AZ s.sherwin(a)imperial.ac.uk<mailto:s.sherwin@imperial.ac.uk> +44 (0) 20 759 45052 Spencer Sherwin McLaren Racing/Royal Academy of Engineering Research Chair, Professor of Computational Fluid Mechanics, Department of Aeronautics, Imperial College London South Kensington Campus London SW7 2AZ s.sherwin(a)imperial.ac.uk<mailto:s.sherwin@imperial.ac.uk> +44 (0) 20 759 45052

1 0

Seg fault with driver adaptive
by Siddarth Chintamani 22 May '16

22 May '16

Hello all, I am tyring to run the IncNavierStokesSolver with adaptive polynomial order. My simulation fails with seg fault as soon as it restarts with adapted polynomial orders. I might be setting my input file wrong. It would be really helpful if someone could point me to an example with adaptive polynomial order. Thank you, Siddarth Chintamani

2 4

EulerCFE - performance issue & MPI problem
by Kühnelt Helmut 20 May '16

20 May '16

Hi Spencer, hi All, I experience that CompressibleFlowSolver for EulerCFE in 1D is really slow. A computation on a grid with 5000 elements, with P=5 and a 5th order RK_SSP (self-implemented) needs ~1 s per time step on an up-to-date workstation (single core execution). (A calculation of 1e5 steps (1s with a time step = 1e-5s) needs 27 hours...) The profiler shows that 68% ot the time the code spends releasing of shared pointers, constructing and destructing Array(), lock() and unlock() and the = operator. Flat profile: Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls ms/call ms/call name 32.05 1.38 1.38 10032 0.14 0.14 boost::detail::sp_counted_base::release() 9.56 1.79 0.41 20514 0.02 0.02 Nektar::Array<Nektar::OneD, double const>::Array(unsigned int, double const&) 9.09 2.18 0.39 7693730 0.00 0.00 Nektar::Array<Nektar::OneD, double const>::~Array() 8.86 2.56 0.38 137658633 0.00 0.00 boost::unique_lock<boost::mutex>::lock() 8.04 2.90 0.35 137608137 0.00 0.00 boost::mutex::unlock() 7.69 3.23 0.33 20528 0.02 0.02 Nektar::Array<Nektar::OneD, double const>::operator=(Nektar::Array<Nektar::OneD, double const> const&) 7.23 3.54 0.31 2500500 0.00 0.00 Nektar::ExactSolverToro::v_PointSolve(double, double, double, double, double, double, double, double, double, double, double&, double&, double&, double&, double&) 4.31 3.73 0.19 74253 0.00 0.01 Nektar::MemPool::Allocate(unsigned long) 2.33 3.83 0.10 Nektar::Array<Nektar::OneD, int const>::~Array() ... Running CompressibleFlowSolver with MPI gives a segmentation fault somewhere in the mesh partitioning (1D mesh) MeshPartition::MeshPartition() MeshPartition::ReadGeometry() 0x15d70d0 0x15d71a0 0x15d71a0 0 [node92:28879] *** Process received signal *** [node92:28879] Signal: Segmentation fault (11) [node92:28879] Signal code: Address not mapped (1) [node92:28879] Failing at address: 0x38 [node92:28879] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f7e7a781d40] [node92:28879] [ 1] /home/hkuehnelt/nektar++/build/library/LibUtilities/libLibUtilities.so.4.3.0(_ZN6Nektar12LibUtilities13MeshPartition12ReadGeometryERKN5boost10shared_ptrINS0_13SessionReaderEEE+0x1268) [0x7f7e7be44118] [node92:28879] [ 2] /home/hkuehnelt/nektar++/build/library/LibUtilities/libLibUtilities.so.4.3.0(_ZN6Nektar12LibUtilities13MeshPartitionC1ERKN5boost10shared_ptrINS0_13SessionReaderEEE+0x40f) [0x7f7e7be44a5f] [node92:28879] [ 3] /home/hkuehnelt/nektar++/build/library/LibUtilities/libLibUtilities.so.4.3.0(_ZN6Nektar12LibUtilities18MeshPartitionMetisC2ERKN5boost10shared_ptrINS0_13SessionReaderEEE+0x17) [0x7f7e7be519f7] [node92:28879] [ 4] /home/hkuehnelt/nektar++/build/library/LibUtilities/libLibUtilities.so.4.3.0(_ZN6Nektar12LibUtilities18MeshPartitionMetis6createERKN5boost10shared_ptrINS0_13SessionReaderEEE+0xc5) [0x7f7e7be53505] [node92:28879] [ 5] /home/hkuehnelt/nektar++/build/library/LibUtilities/libLibUtilities.so.4.3.0(_ZN6Nektar12LibUtilities10NekFactoryISsNS0_13MeshPartitionERKN5boost10shared_ptrINS0_13SessionReaderEEENS0_4noneES9_S9_S9_E14CreateInstanceESsS8_+0x96) [0x7f7e7be77c26] [node92:28879] [ 6] /home/hkuehnelt/nektar++/build/library/LibUtilities/libLibUtilities.so.4.3.0(_ZN6Nektar12LibUtilities13SessionReader13PartitionMeshEv+0x3fd) [0x7f7e7be6cb5d] [node92:28879] [ 7] /home/hkuehnelt/nektar++/build/library/LibUtilities/libLibUtilities.so.4.3.0(_ZN6Nektar12LibUtilities13SessionReader11InitSessionEv+0x55) [0x7f7e7be6de55] [node92:28879] [ 8] /home/hkuehnelt/nektar++/build/solvers/CompressibleFlowSolver/CompressibleFlowSolver(_ZN6Nektar12LibUtilities13SessionReader14CreateInstanceEiPPc+0x14c) [0x4396bc] [node92:28879] [ 9] /home/hkuehnelt/nektar++/build/solvers/CompressibleFlowSolver/CompressibleFlowSolver(main+0x4d) [0x4291ed] [node92:28879] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7e7a76cec5] [node92:28879] [11] /home/hkuehnelt/nektar++/build/solvers/CompressibleFlowSolver/CompressibleFlowSolver() [0x430b99] [node92:28879] *** End of error message *** -------------------------------------------------------------------------- mpirun noticed that process rank 0 with PID 28879 on node node92 exited on signal 11 (Segmentation fault). -------------------------------------------------------------------------- Do you have any advice how to speed up the code and to fix the MPI issue? Best regards, Helmut ________________________________________ Helmut Kühnelt Scientist Mobility Department Electric Drive Technologies AIT Austrian Institute of Technology GmbH Giefinggasse 2 | 1210 Vienna | Austria T: +43 50550-6245 | M: +43 664 815 78 38 | F: +43 50550-6595 helmut.kuehnelt(a)ait.ac.at<https://aitmail.ait.ac.at/owa/redir.aspx?C=LR7CP_NRwEmvpxwWXICJVa6k5FhGic8I…> | http://www.ait.ac.at/<https://aitmail.ait.ac.at/owa/redir.aspx?C=LR7CP_NRwEmvpxwWXICJVa6k5FhGic8I…> FN: 115980 i HG Wien | UID: ATU14703506 This email and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient, please notify the sender by return e-mail or by telephone and delete this message from your system and any printout thereof. Any unauthorized use, reproduction, or dissemination of this message is strictly prohibited. Please note that e-mails are susceptible to change. AIT Austrian Institute of Technology GmbH shall not be liable for the improper or incomplete transmission of the information contained in this communication, nor shall it be liable for any delay in its receipt.

3 14

Fwd: profiling screenshot
by Sherwin, Spencer J 19 May '16

19 May '16

Hi Helmut, Chris ran the 1D solver and profiled it locally. When using Collections with auto-tuning he got the attached profile. If he turned off auto-tuning we did observe the mutex_lock being second highest and dgemv being the top of the list. However in your earlier profiling you had a boost method we do not recognise at the top. Could you perhaps try profiling your run again but with the auto-tuning turned on to see if this boost method is still at the top of your profiling list. If so the discrepancy is presumably related to this method and we should try and find out why this is arising. Cheers, Spencer PS We do have a threading library which has some lock on the memory pool and is probably why the no-auto-tuning is leading to high number of lock. We will look into turning off this feature if threading is not being used PPS Douglas is going to look into using boost 1.57 (since we are generally using 1.58 or higher) to see if he observes a similar slow down to what you see. Begin forwarded message: From: Chris Cantwell <c.cantwell(a)imperial.ac.uk<mailto:c.cantwell@imperial.ac.uk>> Subject: profiling screenshot Date: 18 May 2016 at 18:04:26 BST To: Spencer Sherwin <s.sherwin(a)imperial.ac.uk<mailto:s.sherwin@imperial.ac.uk>> [cid:0BEE0A69-B4F0-4DCC-BA56-28B13FFC4E9A@ae.ic.ac.uk] Spencer Sherwin McLaren Racing/Royal Academy of Engineering Research Chair, Professor of Computational Fluid Mechanics, Department of Aeronautics, Imperial College London South Kensington Campus London SW7 2AZ s.sherwin(a)imperial.ac.uk<mailto:s.sherwin@imperial.ac.uk> +44 (0) 20 759 45052

2 1

Smoothing a specified field
by Jian Yu 06 May '16

06 May '16

Dear all, I am trying to develop a function in the CompressibleFlowSolver defined as void CompressibleFlowSystem::SmoothingFieldGlobal( const Array<OneD, NekDouble > &var, Array<OneD, NekDouble > &var_smooth) The basic process in this function is that 1. First the original field vaule var is projected onto a basis with lower accuracy orders; 2. Averaging is conducted for points along element edges to achieve C0 continuity; 3. Project modified values back to the original basis to obtain var_smooth. Through the above procedures, the specified field value is smoothed to some extent. Has anyone done similar work before or does Nektar++ already contains similar functions? Right now I am not sure where to start.I noticed that there is a C0Projection module in the FieldConvert modules. Is it feasible to transplant this module? Any suggestion would be a big help! Thank you! Best wishes, Jian Yu

1 0