Hi Matthew, Thanks for your advice! I did some basic tests with the Navier-Stokes solver inside the Firedrake. For the lid-driven cavity problem on a NxNxN grid, it takes 23sec for N=5, 594sec for N=10, and 7424sec for N=20. For N=40 it almost takes forever, but it's really not a dense mesh in practice. I had the same experience with my own implementation of fem, using Pardiso to solve the linear system directly, and I've been finding efficient linear solvers to get rid of this problem, so I'm asking if the Firedrake team has any better options for the linear solver. Best, Zizhou On Mon, Apr 12, 2021 at 11:21 PM Matthew Knepley <knepley@gmail.com> wrote:

On Mon, Apr 12, 2021 at 11:00 AM Zizhou Huang <zizhou@nyu.edu> wrote:

...

To one whom it may concern,

I'm a PhD student at NYU. I'm working on a project on solving the Navier-Stokes equation with FEM, so I'm really interested in the amazing demo of solving the Navier-Stokes equation on the website. ( https://www.firedrakeproject.org/demos/navier_stokes.py.html)

I modified the code to solve the 3d lid-driven cavity problem with a uniform grid of size 40x40x40, and found that it takes extremely long time and large memory. I had the same issue with my code of similar sizes, which uses Pardiso to solve the linear system. I'm wondering if fem is indeed unsolvable for a large problem like this? Do you have any experience to make it faster by changing the parameters in the linear solver?

When you want to understand how a solver is scaling, start with a small problem, and look at the performance with -log_view. Then start increasing the problem size, and look at how the number of iterates, subiterates, and event times scale. Once you have that data, getting help is much easier.

Thanks,

Matt

...

Looking forward to hearing from you.

Best, Zizhou _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

I'm using exactly the same solver as this website ( https://www.firedrakeproject.org/demos/navier_stokes.py.html). It seems that they're using fgmres with the PCDPC preconditioner. The logs are as follows. N = 5

0 SNES Function norm 3.873711890948e+00 1 SNES Function norm 7.100383245197e-02 2 SNES Function norm 3.993316741711e-03 3 SNES Function norm 3.990405864913e-03 4 SNES Function norm 3.990405862838e-03 0 SNES Function norm 3.873711890948e+00 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 3.873711890948e+00 true resid norm 3.873711890948e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.922269177394e-02 true resid norm 2.922269177394e-02 ||r(i)||/||b|| 7.543847502500e-03 2 KSP unpreconditioned resid norm 1.837681883991e-03 true resid norm 1.837681883991e-03 ||r(i)||/||b|| 4.743981833769e-04 3 KSP unpreconditioned resid norm 1.837657797525e-03 true resid norm 1.837657797525e-03 ||r(i)||/||b|| 4.743919654476e-04 4 KSP unpreconditioned resid norm 1.837564856663e-03 true resid norm 1.837564856663e-03 ||r(i)||/||b|| 4.743679727336e-04 5 KSP unpreconditioned resid norm 1.837330879480e-03 true resid norm 1.837330880004e-03 ||r(i)||/||b|| 4.743075715821e-04 6 KSP unpreconditioned resid norm 5.396256497007e-04 true resid norm 5.396449573863e-04 ||r(i)||/||b|| 1.393095234179e-04 7 KSP unpreconditioned resid norm 9.179035795321e-07 true resid norm 1.140070292524e-05 ||r(i)||/||b|| 2.943095213631e-06 1 SNES Function norm 7.026786066650e-02 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 7.026786066650e-02 true resid norm 7.026786066650e-02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.995090800849e-02 true resid norm 1.995090800849e-02 ||r(i)||/||b|| 2.839265037993e-01 2 KSP unpreconditioned resid norm 1.115563390429e-03 true resid norm 1.115563390429e-03 ||r(i)||/||b|| 1.587586956323e-02 3 KSP unpreconditioned resid norm 1.115554835302e-03 true resid norm 1.115554835302e-03 ||r(i)||/||b|| 1.587574781302e-02 4 KSP unpreconditioned resid norm 1.115554764006e-03 true resid norm 1.115554764006e-03 ||r(i)||/||b|| 1.587574679839e-02 5 KSP unpreconditioned resid norm 1.115548853894e-03 true resid norm 1.115548853906e-03 ||r(i)||/||b|| 1.587566269024e-02 6 KSP unpreconditioned resid norm 7.632184921082e-05 true resid norm 7.632265261003e-05 ||r(i)||/||b|| 1.086167301610e-03 7 KSP unpreconditioned resid norm 1.422983940334e-08 true resid norm 4.141122043271e-08 ||r(i)||/||b|| 5.893337300996e-07 2 SNES Function norm 8.778069005998e-06 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 8.778069005998e-06 true resid norm 8.778069005998e-06 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 3.525945734318e-06 true resid norm 3.525945734318e-06 ||r(i)||/||b|| 4.016766935768e-01 2 KSP unpreconditioned resid norm 7.092855329067e-07 true resid norm 7.092855329067e-07 ||r(i)||/||b|| 8.080200012349e-02 3 KSP unpreconditioned resid norm 7.092827495949e-07 true resid norm 7.092827495949e-07 ||r(i)||/||b|| 8.080168304786e-02 4 KSP unpreconditioned resid norm 7.092826142571e-07 true resid norm 7.092826142574e-07 ||r(i)||/||b|| 8.080166763017e-02 5 KSP unpreconditioned resid norm 8.490734432004e-08 true resid norm 8.490728815768e-08 ||r(i)||/||b|| 9.672661276604e-03 6 KSP unpreconditioned resid norm 3.681976478361e-12 true resid norm 9.310211086004e-12 ||r(i)||/||b|| 1.060621769963e-06 3 SNES Function norm 9.754530371801e-12

N = 10

0 SNES Function norm 3.857209122044e+00 1 SNES Function norm 2.174942016837e-02 2 SNES Function norm 7.117856370756e-04 3 SNES Function norm 7.102760546744e-04 4 SNES Function norm 7.102760536032e-04 0 SNES Function norm 3.857209122044e+00 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 3.857209122044e+00 true resid norm 3.857209122044e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.386167872908e-01 true resid norm 1.386167872908e-01 ||r(i)||/||b|| 3.593706820265e-02 2 KSP unpreconditioned resid norm 4.475586255809e-02 true resid norm 4.475586255809e-02 ||r(i)||/||b|| 1.160317243426e-02 3 KSP unpreconditioned resid norm 1.722695448041e-02 true resid norm 1.722695448041e-02 ||r(i)||/||b|| 4.466170730012e-03 4 KSP unpreconditioned resid norm 5.231768306662e-03 true resid norm 5.231768306662e-03 ||r(i)||/||b|| 1.356361073804e-03 5 KSP unpreconditioned resid norm 2.041434254281e-03 true resid norm 2.041434254281e-03 ||r(i)||/||b|| 5.292516401597e-04 6 KSP unpreconditioned resid norm 5.875484274903e-04 true resid norm 5.875484274903e-04 ||r(i)||/||b|| 1.523247531829e-04 7 KSP unpreconditioned resid norm 2.354394338936e-04 true resid norm 2.354394338936e-04 ||r(i)||/||b|| 6.103880459787e-05 8 KSP unpreconditioned resid norm 1.414423189101e-04 true resid norm 1.414423189101e-04 ||r(i)||/||b|| 3.666960085254e-05 9 KSP unpreconditioned resid norm 1.272526161323e-04 true resid norm 1.272526161323e-04 ||r(i)||/||b|| 3.299085222139e-05 10 KSP unpreconditioned resid norm 1.256180185176e-04 true resid norm 1.256180185176e-04 ||r(i)||/||b|| 3.256707493500e-05 11 KSP unpreconditioned resid norm 1.253360959842e-04 true resid norm 1.253360959842e-04 ||r(i)||/||b|| 3.249398516348e-05 12 KSP unpreconditioned resid norm 1.252954886770e-04 true resid norm 1.252954886770e-04 ||r(i)||/||b|| 3.248345752397e-05 13 KSP unpreconditioned resid norm 1.252778828690e-04 true resid norm 1.252778828690e-04 ||r(i)||/||b|| 3.247889313364e-05 14 KSP unpreconditioned resid norm 1.252729271101e-04 true resid norm 1.252729271101e-04 ||r(i)||/||b|| 3.247760832934e-05 15 KSP unpreconditioned resid norm 1.252709541672e-04 true resid norm 1.252709541672e-04 ||r(i)||/||b|| 3.247709683442e-05 16 KSP unpreconditioned resid norm 1.252698774325e-04 true resid norm 1.252698774325e-04 ||r(i)||/||b|| 3.247681768576e-05 17 KSP unpreconditioned resid norm 1.252693026661e-04 true resid norm 1.252693026661e-04 ||r(i)||/||b|| 3.247666867482e-05 18 KSP unpreconditioned resid norm 1.252691813938e-04 true resid norm 1.252691813938e-04 ||r(i)||/||b|| 3.247663723438e-05 19 KSP unpreconditioned resid norm 1.252690631661e-04 true resid norm 1.252690631661e-04 ||r(i)||/||b|| 3.247660658329e-05 20 KSP unpreconditioned resid norm 1.252682040589e-04 true resid norm 1.252682040589e-04 ||r(i)||/||b|| 3.247638385562e-05 21 KSP unpreconditioned resid norm 1.252666235238e-04 true resid norm 1.252666235237e-04 ||r(i)||/||b|| 3.247597409427e-05 22 KSP unpreconditioned resid norm 1.252663551040e-04 true resid norm 1.252663551038e-04 ||r(i)||/||b|| 3.247590450512e-05 23 KSP unpreconditioned resid norm 1.252657099958e-04 true resid norm 1.252657099961e-04 ||r(i)||/||b|| 3.247573725784e-05 24 KSP unpreconditioned resid norm 1.252222519823e-04 true resid norm 1.252222519982e-04 ||r(i)||/||b|| 3.246447056306e-05 25 KSP unpreconditioned resid norm 1.248107616721e-04 true resid norm 1.248107618126e-04 ||r(i)||/||b|| 3.235778975510e-05 26 KSP unpreconditioned resid norm 1.197171374954e-04 true resid norm 1.197171390035e-04 ||r(i)||/||b|| 3.103724356537e-05 27 KSP unpreconditioned resid norm 9.324077689908e-05 true resid norm 9.324078248863e-05 ||r(i)||/||b|| 2.417312091163e-05 28 KSP unpreconditioned resid norm 3.902477473206e-05 true resid norm 3.902477892599e-05 ||r(i)||/||b|| 1.011736146297e-05 29 KSP unpreconditioned resid norm 1.307801475003e-05 true resid norm 1.307801741838e-05 ||r(i)||/||b|| 3.390538859724e-06 1 SNES Function norm 2.171186815034e-02 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 2.171186815034e-02 true resid norm 2.171186815034e-02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 5.755446049476e-03 true resid norm 5.755446049476e-03 ||r(i)||/||b|| 2.650829495474e-01 2 KSP unpreconditioned resid norm 1.819729526723e-03 true resid norm 1.819729526723e-03 ||r(i)||/||b|| 8.381266476578e-02 3 KSP unpreconditioned resid norm 5.868270016474e-04 true resid norm 5.868270016474e-04 ||r(i)||/||b|| 2.702793686771e-02 4 KSP unpreconditioned resid norm 1.891824694427e-04 true resid norm 1.891824694427e-04 ||r(i)||/||b|| 8.713320665581e-03 5 KSP unpreconditioned resid norm 7.091162609666e-05 true resid norm 7.091162609666e-05 ||r(i)||/||b|| 3.266030615406e-03 6 KSP unpreconditioned resid norm 5.637701208830e-05 true resid norm 5.637701208830e-05 ||r(i)||/||b|| 2.596598860030e-03 7 KSP unpreconditioned resid norm 5.383999628854e-05 true resid norm 5.383999628854e-05 ||r(i)||/||b|| 2.479749596660e-03 8 KSP unpreconditioned resid norm 5.369249277053e-05 true resid norm 5.369249277053e-05 ||r(i)||/||b|| 2.472955915113e-03 9 KSP unpreconditioned resid norm 5.368343313838e-05 true resid norm 5.368343313838e-05 ||r(i)||/||b|| 2.472538648755e-03 10 KSP unpreconditioned resid norm 5.368324507663e-05 true resid norm 5.368324507663e-05 ||r(i)||/||b|| 2.472529987052e-03 11 KSP unpreconditioned resid norm 5.367688316238e-05 true resid norm 5.367688316238e-05 ||r(i)||/||b|| 2.472236971536e-03 12 KSP unpreconditioned resid norm 5.364660764375e-05 true resid norm 5.364660764375e-05 ||r(i)||/||b|| 2.470842548982e-03 13 KSP unpreconditioned resid norm 5.363778445858e-05 true resid norm 5.363778445858e-05 ||r(i)||/||b|| 2.470436172843e-03 14 KSP unpreconditioned resid norm 5.363146760267e-05 true resid norm 5.363146760267e-05 ||r(i)||/||b|| 2.470145232613e-03 15 KSP unpreconditioned resid norm 5.363036199206e-05 true resid norm 5.363036199206e-05 ||r(i)||/||b|| 2.470094310665e-03 16 KSP unpreconditioned resid norm 5.362586041467e-05 true resid norm 5.362586041467e-05 ||r(i)||/||b|| 2.469886978097e-03 17 KSP unpreconditioned resid norm 5.362298140611e-05 true resid norm 5.362298140610e-05 ||r(i)||/||b|| 2.469754377413e-03 18 KSP unpreconditioned resid norm 5.362041508854e-05 true resid norm 5.362041508851e-05 ||r(i)||/||b|| 2.469636178574e-03 19 KSP unpreconditioned resid norm 5.362023682045e-05 true resid norm 5.362023682040e-05 ||r(i)||/||b|| 2.469627967945e-03 20 KSP unpreconditioned resid norm 5.362021431482e-05 true resid norm 5.362021431475e-05 ||r(i)||/||b|| 2.469626931385e-03 21 KSP unpreconditioned resid norm 5.362021307632e-05 true resid norm 5.362021307624e-05 ||r(i)||/||b|| 2.469626874342e-03 22 KSP unpreconditioned resid norm 5.362020945699e-05 true resid norm 5.362020945697e-05 ||r(i)||/||b|| 2.469626707646e-03 23 KSP unpreconditioned resid norm 5.360919215998e-05 true resid norm 5.360919217038e-05 ||r(i)||/||b|| 2.469119276111e-03 24 KSP unpreconditioned resid norm 5.352778280091e-05 true resid norm 5.352778288656e-05 ||r(i)||/||b|| 2.465369746902e-03 25 KSP unpreconditioned resid norm 5.256642319528e-05 true resid norm 5.256642407158e-05 ||r(i)||/||b|| 2.421091713877e-03 26 KSP unpreconditioned resid norm 4.729160583219e-05 true resid norm 4.729161069155e-05 ||r(i)||/||b|| 2.178145628192e-03 27 KSP unpreconditioned resid norm 2.668806077499e-05 true resid norm 2.668806954502e-05 ||r(i)||/||b|| 1.229192686701e-03 28 KSP unpreconditioned resid norm 1.020558332703e-05 true resid norm 1.020558761616e-05 ||r(i)||/||b|| 4.700464992463e-04 29 KSP unpreconditioned resid norm 3.451738103984e-06 true resid norm 3.451741888987e-06 ||r(i)||/||b|| 1.589794975304e-04 30 KSP unpreconditioned resid norm 1.120801314646e-06 true resid norm 1.120801314646e-06 ||r(i)||/||b|| 5.162159731651e-05 31 KSP unpreconditioned resid norm 6.657758394387e-07 true resid norm 6.657758394387e-07 ||r(i)||/||b|| 3.066414344582e-05 32 KSP unpreconditioned resid norm 1.791100894652e-07 true resid norm 1.791100894652e-07 ||r(i)||/||b|| 8.249409411708e-06 2 SNES Function norm 1.208714436027e-06 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 1.208714436027e-06 true resid norm 1.208714436027e-06 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 7.009704990544e-07 true resid norm 7.009704990544e-07 ||r(i)||/||b|| 5.799306090514e-01 2 KSP unpreconditioned resid norm 2.928942580645e-07 true resid norm 2.928942580645e-07 ||r(i)||/||b|| 2.423188218279e-01 3 KSP unpreconditioned resid norm 1.900700846501e-07 true resid norm 1.900700846501e-07 ||r(i)||/||b|| 1.572497845520e-01 4 KSP unpreconditioned resid norm 1.694415211872e-07 true resid norm 1.694415211872e-07 ||r(i)||/||b|| 1.401832526665e-01 5 KSP unpreconditioned resid norm 1.670108100598e-07 true resid norm 1.670108100598e-07 ||r(i)||/||b|| 1.381722639210e-01 6 KSP unpreconditioned resid norm 1.668707304029e-07 true resid norm 1.668707304029e-07 ||r(i)||/||b|| 1.380563724807e-01 7 KSP unpreconditioned resid norm 1.668026114154e-07 true resid norm 1.668026114154e-07 ||r(i)||/||b|| 1.380000159208e-01 8 KSP unpreconditioned resid norm 1.668025617944e-07 true resid norm 1.668025617944e-07 ||r(i)||/||b|| 1.379999748681e-01 9 KSP unpreconditioned resid norm 1.667959318665e-07 true resid norm 1.667959318665e-07 ||r(i)||/||b|| 1.379944897612e-01 10 KSP unpreconditioned resid norm 1.667495881623e-07 true resid norm 1.667495881623e-07 ||r(i)||/||b|| 1.379561484435e-01 11 KSP unpreconditioned resid norm 1.667472366297e-07 true resid norm 1.667472366297e-07 ||r(i)||/||b|| 1.379542029611e-01 12 KSP unpreconditioned resid norm 1.667421748942e-07 true resid norm 1.667421748942e-07 ||r(i)||/||b|| 1.379500152595e-01 13 KSP unpreconditioned resid norm 1.667405719095e-07 true resid norm 1.667405719095e-07 ||r(i)||/||b|| 1.379486890697e-01 14 KSP unpreconditioned resid norm 1.667313112753e-07 true resid norm 1.667313112754e-07 ||r(i)||/||b|| 1.379410275130e-01 15 KSP unpreconditioned resid norm 1.667263791233e-07 true resid norm 1.667263791233e-07 ||r(i)||/||b|| 1.379369470190e-01 16 KSP unpreconditioned resid norm 1.666960300587e-07 true resid norm 1.666960300590e-07 ||r(i)||/||b|| 1.379118384711e-01 17 KSP unpreconditioned resid norm 1.666924323014e-07 true resid norm 1.666924323014e-07 ||r(i)||/||b|| 1.379088619553e-01 18 KSP unpreconditioned resid norm 1.666835606480e-07 true resid norm 1.666835606492e-07 ||r(i)||/||b|| 1.379015222132e-01 19 KSP unpreconditioned resid norm 1.666784002237e-07 true resid norm 1.666784002274e-07 ||r(i)||/||b|| 1.378972528659e-01 20 KSP unpreconditioned resid norm 1.665390347333e-07 true resid norm 1.665390347759e-07 ||r(i)||/||b|| 1.377819523057e-01 21 KSP unpreconditioned resid norm 1.662598119876e-07 true resid norm 1.662598121924e-07 ||r(i)||/||b|| 1.375509444058e-01 22 KSP unpreconditioned resid norm 1.625064404254e-07 true resid norm 1.625064423607e-07 ||r(i)||/||b|| 1.344456866875e-01 23 KSP unpreconditioned resid norm 1.391707373331e-07 true resid norm 1.391707472513e-07 ||r(i)||/||b|| 1.151394763752e-01 24 KSP unpreconditioned resid norm 7.505751761619e-08 true resid norm 7.505754230672e-08 ||r(i)||/||b|| 6.209700163211e-02 25 KSP unpreconditioned resid norm 2.799964315999e-08 true resid norm 2.799965549375e-08 ||r(i)||/||b|| 2.316482260755e-02 26 KSP unpreconditioned resid norm 9.970551878855e-09 true resid norm 9.970561618435e-09 ||r(i)||/||b|| 8.248897606624e-03 27 KSP unpreconditioned resid norm 3.252551635867e-09 true resid norm 3.252558544160e-09 ||r(i)||/||b|| 2.690923883437e-03 28 KSP unpreconditioned resid norm 1.136728449942e-09 true resid norm 1.136739747171e-09 ||r(i)||/||b|| 9.404535209387e-04 29 KSP unpreconditioned resid norm 3.672093573536e-10 true resid norm 3.672155322102e-10 ||r(i)||/||b|| 3.038066902032e-04 30 KSP unpreconditioned resid norm 1.234864719987e-10 true resid norm 1.234864719987e-10 ||r(i)||/||b|| 1.021634790800e-04 31 KSP unpreconditioned resid norm 6.510725158420e-11 true resid norm 6.510725158421e-11 ||r(i)||/||b|| 5.386487464998e-05 32 KSP unpreconditioned resid norm 1.805042777062e-11 true resid norm 1.805042777059e-11 ||r(i)||/||b|| 1.493357507164e-05 33 KSP unpreconditioned resid norm 7.492522181774e-12 true resid norm 7.492522181758e-12 ||r(i)||/||b|| 6.198752954739e-06 3 SNES Function norm 7.512220214011e-12

N=20

0 SNES Function norm 3.857209122044e+00 1 SNES Function norm 2.174942016837e-02 2 SNES Function norm 7.117856370756e-04 3 SNES Function norm 7.102760546744e-04 4 SNES Function norm 7.102760536032e-04 0 SNES Function norm 3.857209122044e+00 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 3.857209122044e+00 true resid norm 3.857209122044e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.386167872908e-01 true resid norm 1.386167872908e-01 ||r(i)||/||b|| 3.593706820265e-02 2 KSP unpreconditioned resid norm 4.475586255809e-02 true resid norm 4.475586255809e-02 ||r(i)||/||b|| 1.160317243426e-02 3 KSP unpreconditioned resid norm 1.722695448041e-02 true resid norm 1.722695448041e-02 ||r(i)||/||b|| 4.466170730012e-03 4 KSP unpreconditioned resid norm 5.231768306662e-03 true resid norm 5.231768306662e-03 ||r(i)||/||b|| 1.356361073804e-03 5 KSP unpreconditioned resid norm 2.041434254281e-03 true resid norm 2.041434254281e-03 ||r(i)||/||b|| 5.292516401597e-04 6 KSP unpreconditioned resid norm 5.875484274903e-04 true resid norm 5.875484274903e-04 ||r(i)||/||b|| 1.523247531829e-04 7 KSP unpreconditioned resid norm 2.354394338936e-04 true resid norm 2.354394338936e-04 ||r(i)||/||b|| 6.103880459787e-05 8 KSP unpreconditioned resid norm 1.414423189101e-04 true resid norm 1.414423189101e-04 ||r(i)||/||b|| 3.666960085254e-05 9 KSP unpreconditioned resid norm 1.272526161323e-04 true resid norm 1.272526161323e-04 ||r(i)||/||b|| 3.299085222139e-05 10 KSP unpreconditioned resid norm 1.256180185176e-04 true resid norm 1.256180185176e-04 ||r(i)||/||b|| 3.256707493500e-05 11 KSP unpreconditioned resid norm 1.253360959842e-04 true resid norm 1.253360959842e-04 ||r(i)||/||b|| 3.249398516348e-05 12 KSP unpreconditioned resid norm 1.252954886770e-04 true resid norm 1.252954886770e-04 ||r(i)||/||b|| 3.248345752397e-05 13 KSP unpreconditioned resid norm 1.252778828690e-04 true resid norm 1.252778828690e-04 ||r(i)||/||b|| 3.247889313364e-05 14 KSP unpreconditioned resid norm 1.252729271101e-04 true resid norm 1.252729271101e-04 ||r(i)||/||b|| 3.247760832934e-05 15 KSP unpreconditioned resid norm 1.252709541672e-04 true resid norm 1.252709541672e-04 ||r(i)||/||b|| 3.247709683442e-05 16 KSP unpreconditioned resid norm 1.252698774325e-04 true resid norm 1.252698774325e-04 ||r(i)||/||b|| 3.247681768576e-05 17 KSP unpreconditioned resid norm 1.252693026661e-04 true resid norm 1.252693026661e-04 ||r(i)||/||b|| 3.247666867482e-05 18 KSP unpreconditioned resid norm 1.252691813938e-04 true resid norm 1.252691813938e-04 ||r(i)||/||b|| 3.247663723438e-05 19 KSP unpreconditioned resid norm 1.252690631661e-04 true resid norm 1.252690631661e-04 ||r(i)||/||b|| 3.247660658329e-05 20 KSP unpreconditioned resid norm 1.252682040589e-04 true resid norm 1.252682040589e-04 ||r(i)||/||b|| 3.247638385562e-05 21 KSP unpreconditioned resid norm 1.252666235238e-04 true resid norm 1.252666235237e-04 ||r(i)||/||b|| 3.247597409427e-05 22 KSP unpreconditioned resid norm 1.252663551040e-04 true resid norm 1.252663551038e-04 ||r(i)||/||b|| 3.247590450512e-05 23 KSP unpreconditioned resid norm 1.252657099958e-04 true resid norm 1.252657099961e-04 ||r(i)||/||b|| 3.247573725784e-05 24 KSP unpreconditioned resid norm 1.252222519823e-04 true resid norm 1.252222519982e-04 ||r(i)||/||b|| 3.246447056306e-05 25 KSP unpreconditioned resid norm 1.248107616721e-04 true resid norm 1.248107618126e-04 ||r(i)||/||b|| 3.235778975510e-05 26 KSP unpreconditioned resid norm 1.197171374954e-04 true resid norm 1.197171390035e-04 ||r(i)||/||b|| 3.103724356537e-05 27 KSP unpreconditioned resid norm 9.324077689908e-05 true resid norm 9.324078248863e-05 ||r(i)||/||b|| 2.417312091163e-05 28 KSP unpreconditioned resid norm 3.902477473206e-05 true resid norm 3.902477892599e-05 ||r(i)||/||b|| 1.011736146297e-05 29 KSP unpreconditioned resid norm 1.307801475003e-05 true resid norm 1.307801741838e-05 ||r(i)||/||b|| 3.390538859724e-06 1 SNES Function norm 2.171186815034e-02 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 2.171186815034e-02 true resid norm 2.171186815034e-02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 5.755446049476e-03 true resid norm 5.755446049476e-03 ||r(i)||/||b|| 2.650829495474e-01 2 KSP unpreconditioned resid norm 1.819729526723e-03 true resid norm 1.819729526723e-03 ||r(i)||/||b|| 8.381266476578e-02 3 KSP unpreconditioned resid norm 5.868270016474e-04 true resid norm 5.868270016474e-04 ||r(i)||/||b|| 2.702793686771e-02 4 KSP unpreconditioned resid norm 1.891824694427e-04 true resid norm 1.891824694427e-04 ||r(i)||/||b|| 8.713320665581e-03 5 KSP unpreconditioned resid norm 7.091162609666e-05 true resid norm 7.091162609666e-05 ||r(i)||/||b|| 3.266030615406e-03 6 KSP unpreconditioned resid norm 5.637701208830e-05 true resid norm 5.637701208830e-05 ||r(i)||/||b|| 2.596598860030e-03 7 KSP unpreconditioned resid norm 5.383999628854e-05 true resid norm 5.383999628854e-05 ||r(i)||/||b|| 2.479749596660e-03 8 KSP unpreconditioned resid norm 5.369249277053e-05 true resid norm 5.369249277053e-05 ||r(i)||/||b|| 2.472955915113e-03 9 KSP unpreconditioned resid norm 5.368343313838e-05 true resid norm 5.368343313838e-05 ||r(i)||/||b|| 2.472538648755e-03 10 KSP unpreconditioned resid norm 5.368324507663e-05 true resid norm 5.368324507663e-05 ||r(i)||/||b|| 2.472529987052e-03 11 KSP unpreconditioned resid norm 5.367688316238e-05 true resid norm 5.367688316238e-05 ||r(i)||/||b|| 2.472236971536e-03 12 KSP unpreconditioned resid norm 5.364660764375e-05 true resid norm 5.364660764375e-05 ||r(i)||/||b|| 2.470842548982e-03 13 KSP unpreconditioned resid norm 5.363778445858e-05 true resid norm 5.363778445858e-05 ||r(i)||/||b|| 2.470436172843e-03 14 KSP unpreconditioned resid norm 5.363146760267e-05 true resid norm 5.363146760267e-05 ||r(i)||/||b|| 2.470145232613e-03 15 KSP unpreconditioned resid norm 5.363036199206e-05 true resid norm 5.363036199206e-05 ||r(i)||/||b|| 2.470094310665e-03 16 KSP unpreconditioned resid norm 5.362586041467e-05 true resid norm 5.362586041467e-05 ||r(i)||/||b|| 2.469886978097e-03 17 KSP unpreconditioned resid norm 5.362298140611e-05 true resid norm 5.362298140610e-05 ||r(i)||/||b|| 2.469754377413e-03 18 KSP unpreconditioned resid norm 5.362041508854e-05 true resid norm 5.362041508851e-05 ||r(i)||/||b|| 2.469636178574e-03 19 KSP unpreconditioned resid norm 5.362023682045e-05 true resid norm 5.362023682040e-05 ||r(i)||/||b|| 2.469627967945e-03 20 KSP unpreconditioned resid norm 5.362021431482e-05 true resid norm 5.362021431475e-05 ||r(i)||/||b|| 2.469626931385e-03 21 KSP unpreconditioned resid norm 5.362021307632e-05 true resid norm 5.362021307624e-05 ||r(i)||/||b|| 2.469626874342e-03 22 KSP unpreconditioned resid norm 5.362020945699e-05 true resid norm 5.362020945697e-05 ||r(i)||/||b|| 2.469626707646e-03 23 KSP unpreconditioned resid norm 5.360919215998e-05 true resid norm 5.360919217038e-05 ||r(i)||/||b|| 2.469119276111e-03 24 KSP unpreconditioned resid norm 5.352778280091e-05 true resid norm 5.352778288656e-05 ||r(i)||/||b|| 2.465369746902e-03 25 KSP unpreconditioned resid norm 5.256642319528e-05 true resid norm 5.256642407158e-05 ||r(i)||/||b|| 2.421091713877e-03 26 KSP unpreconditioned resid norm 4.729160583219e-05 true resid norm 4.729161069155e-05 ||r(i)||/||b|| 2.178145628192e-03 27 KSP unpreconditioned resid norm 2.668806077499e-05 true resid norm 2.668806954502e-05 ||r(i)||/||b|| 1.229192686701e-03 28 KSP unpreconditioned resid norm 1.020558332703e-05 true resid norm 1.020558761616e-05 ||r(i)||/||b|| 4.700464992463e-04 29 KSP unpreconditioned resid norm 3.451738103984e-06 true resid norm 3.451741888987e-06 ||r(i)||/||b|| 1.589794975304e-04 30 KSP unpreconditioned resid norm 1.120801314646e-06 true resid norm 1.120801314646e-06 ||r(i)||/||b|| 5.162159731651e-05 31 KSP unpreconditioned resid norm 6.657758394387e-07 true resid norm 6.657758394387e-07 ||r(i)||/||b|| 3.066414344582e-05 32 KSP unpreconditioned resid norm 1.791100894652e-07 true resid norm 1.791100894652e-07 ||r(i)||/||b|| 8.249409411708e-06 2 SNES Function norm 1.208714436027e-06 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 1.208714436027e-06 true resid norm 1.208714436027e-06 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 7.009704990544e-07 true resid norm 7.009704990544e-07 ||r(i)||/||b|| 5.799306090514e-01 2 KSP unpreconditioned resid norm 2.928942580645e-07 true resid norm 2.928942580645e-07 ||r(i)||/||b|| 2.423188218279e-01 3 KSP unpreconditioned resid norm 1.900700846501e-07 true resid norm 1.900700846501e-07 ||r(i)||/||b|| 1.572497845520e-01 4 KSP unpreconditioned resid norm 1.694415211872e-07 true resid norm 1.694415211872e-07 ||r(i)||/||b|| 1.401832526665e-01 5 KSP unpreconditioned resid norm 1.670108100598e-07 true resid norm 1.670108100598e-07 ||r(i)||/||b|| 1.381722639210e-01 6 KSP unpreconditioned resid norm 1.668707304029e-07 true resid norm 1.668707304029e-07 ||r(i)||/||b|| 1.380563724807e-01 7 KSP unpreconditioned resid norm 1.668026114154e-07 true resid norm 1.668026114154e-07 ||r(i)||/||b|| 1.380000159208e-01 8 KSP unpreconditioned resid norm 1.668025617944e-07 true resid norm 1.668025617944e-07 ||r(i)||/||b|| 1.379999748681e-01 9 KSP unpreconditioned resid norm 1.667959318665e-07 true resid norm 1.667959318665e-07 ||r(i)||/||b|| 1.379944897612e-01 10 KSP unpreconditioned resid norm 1.667495881623e-07 true resid norm 1.667495881623e-07 ||r(i)||/||b|| 1.379561484435e-01 11 KSP unpreconditioned resid norm 1.667472366297e-07 true resid norm 1.667472366297e-07 ||r(i)||/||b|| 1.379542029611e-01 12 KSP unpreconditioned resid norm 1.667421748942e-07 true resid norm 1.667421748942e-07 ||r(i)||/||b|| 1.379500152595e-01 13 KSP unpreconditioned resid norm 1.667405719095e-07 true resid norm 1.667405719095e-07 ||r(i)||/||b|| 1.379486890697e-01 14 KSP unpreconditioned resid norm 1.667313112753e-07 true resid norm 1.667313112754e-07 ||r(i)||/||b|| 1.379410275130e-01 15 KSP unpreconditioned resid norm 1.667263791233e-07 true resid norm 1.667263791233e-07 ||r(i)||/||b|| 1.379369470190e-01 16 KSP unpreconditioned resid norm 1.666960300587e-07 true resid norm 1.666960300590e-07 ||r(i)||/||b|| 1.379118384711e-01 17 KSP unpreconditioned resid norm 1.666924323014e-07 true resid norm 1.666924323014e-07 ||r(i)||/||b|| 1.379088619553e-01 18 KSP unpreconditioned resid norm 1.666835606480e-07 true resid norm 1.666835606492e-07 ||r(i)||/||b|| 1.379015222132e-01 19 KSP unpreconditioned resid norm 1.666784002237e-07 true resid norm 1.666784002274e-07 ||r(i)||/||b|| 1.378972528659e-01 20 KSP unpreconditioned resid norm 1.665390347333e-07 true resid norm 1.665390347759e-07 ||r(i)||/||b|| 1.377819523057e-01 21 KSP unpreconditioned resid norm 1.662598119876e-07 true resid norm 1.662598121924e-07 ||r(i)||/||b|| 1.375509444058e-01 22 KSP unpreconditioned resid norm 1.625064404254e-07 true resid norm 1.625064423607e-07 ||r(i)||/||b|| 1.344456866875e-01 23 KSP unpreconditioned resid norm 1.391707373331e-07 true resid norm 1.391707472513e-07 ||r(i)||/||b|| 1.151394763752e-01 24 KSP unpreconditioned resid norm 7.505751761619e-08 true resid norm 7.505754230672e-08 ||r(i)||/||b|| 6.209700163211e-02 25 KSP unpreconditioned resid norm 2.799964315999e-08 true resid norm 2.799965549375e-08 ||r(i)||/||b|| 2.316482260755e-02 26 KSP unpreconditioned resid norm 9.970551878855e-09 true resid norm 9.970561618435e-09 ||r(i)||/||b|| 8.248897606624e-03 27 KSP unpreconditioned resid norm 3.252551635867e-09 true resid norm 3.252558544160e-09 ||r(i)||/||b|| 2.690923883437e-03 28 KSP unpreconditioned resid norm 1.136728449942e-09 true resid norm 1.136739747171e-09 ||r(i)||/||b|| 9.404535209387e-04 29 KSP unpreconditioned resid norm 3.672093573536e-10 true resid norm 3.672155322102e-10 ||r(i)||/||b|| 3.038066902032e-04 30 KSP unpreconditioned resid norm 1.234864719987e-10 true resid norm 1.234864719987e-10 ||r(i)||/||b|| 1.021634790800e-04 31 KSP unpreconditioned resid norm 6.510725158420e-11 true resid norm 6.510725158421e-11 ||r(i)||/||b|| 5.386487464998e-05 32 KSP unpreconditioned resid norm 1.805042777062e-11 true resid norm 1.805042777059e-11 ||r(i)||/||b|| 1.493357507164e-05 33 KSP unpreconditioned resid norm 7.492522181774e-12 true resid norm 7.492522181758e-12 ||r(i)||/||b|| 6.198752954739e-06 3 SNES Function norm 7.512220214011e-12

Best, Zizhou On Tue, Apr 13, 2021 at 10:04 AM Matthew Knepley <knepley@gmail.com> wrote:

On Mon, Apr 12, 2021 at 9:45 PM Zizhou Huang <zizhou@nyu.edu> wrote:

...

Hi Matthew,

Thanks for your advice!

I did some basic tests with the Navier-Stokes solver inside the Firedrake. For the lid-driven cavity problem on a NxNxN grid, it takes 23sec for N=5, 594sec for N=10, and 7424sec for N=20. For N=40 it almost takes forever, but it's really not a dense mesh in practice. I had the same experience with my own implementation of fem, using Pardiso to solve the linear system directly, and I've been finding efficient linear solvers to get rid of this problem, so I'm asking if the Firedrake team has any better options for the linear solver.

What solver did you use and how many iterates did it take? Also, posting the logs makes things easier.

Thanks

Matt

...

Best, Zizhou

On Mon, Apr 12, 2021 at 11:21 PM Matthew Knepley <knepley@gmail.com> wrote:

...

On Mon, Apr 12, 2021 at 11:00 AM Zizhou Huang <zizhou@nyu.edu> wrote:

...

To one whom it may concern,

I'm a PhD student at NYU. I'm working on a project on solving the Navier-Stokes equation with FEM, so I'm really interested in the amazing demo of solving the Navier-Stokes equation on the website. ( https://www.firedrakeproject.org/demos/navier_stokes.py.html)

I modified the code to solve the 3d lid-driven cavity problem with a uniform grid of size 40x40x40, and found that it takes extremely long time and large memory. I had the same issue with my code of similar sizes, which uses Pardiso to solve the linear system. I'm wondering if fem is indeed unsolvable for a large problem like this? Do you have any experience to make it faster by changing the parameters in the linear solver?

When you want to understand how a solver is scaling, start with a small problem, and look at the performance with -log_view. Then start increasing the problem size, and look at how the number of iterates, subiterates, and event times scale. Once you have that data, getting help is much easier.

Thanks,

Matt

...

Looking forward to hearing from you.

Best, Zizhou _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Hi Zizhou, That page uses lu for all the inner solves, because it’s just a small-scale demo. You almost certainly want to use something more scalable than that. From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Zizhou Huang <zizhou@nyu.edu> Date: Tuesday, 13 April 2021 at 03:14 To: Matthew Knepley <knepley@gmail.com> Cc: firedrake <firedrake@imperial.ac.uk> Subject: Re: [firedrake] Question about the efficiency of the Navier-Stokes solver in the Firedrake I'm using exactly the same solver as this website (https://www.firedrakeproject.org/demos/navier_stokes.py.html). It seems that they're using fgmres with the PCDPC preconditioner. The logs are as follows. N = 5 0 SNES Function norm 3.873711890948e+00 1 SNES Function norm 7.100383245197e-02 2 SNES Function norm 3.993316741711e-03 3 SNES Function norm 3.990405864913e-03 4 SNES Function norm 3.990405862838e-03 0 SNES Function norm 3.873711890948e+00 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 3.873711890948e+00 true resid norm 3.873711890948e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.922269177394e-02 true resid norm 2.922269177394e-02 ||r(i)||/||b|| 7.543847502500e-03 2 KSP unpreconditioned resid norm 1.837681883991e-03 true resid norm 1.837681883991e-03 ||r(i)||/||b|| 4.743981833769e-04 3 KSP unpreconditioned resid norm 1.837657797525e-03 true resid norm 1.837657797525e-03 ||r(i)||/||b|| 4.743919654476e-04 4 KSP unpreconditioned resid norm 1.837564856663e-03 true resid norm 1.837564856663e-03 ||r(i)||/||b|| 4.743679727336e-04 5 KSP unpreconditioned resid norm 1.837330879480e-03 true resid norm 1.837330880004e-03 ||r(i)||/||b|| 4.743075715821e-04 6 KSP unpreconditioned resid norm 5.396256497007e-04 true resid norm 5.396449573863e-04 ||r(i)||/||b|| 1.393095234179e-04 7 KSP unpreconditioned resid norm 9.179035795321e-07 true resid norm 1.140070292524e-05 ||r(i)||/||b|| 2.943095213631e-06 1 SNES Function norm 7.026786066650e-02 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 7.026786066650e-02 true resid norm 7.026786066650e-02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.995090800849e-02 true resid norm 1.995090800849e-02 ||r(i)||/||b|| 2.839265037993e-01 2 KSP unpreconditioned resid norm 1.115563390429e-03 true resid norm 1.115563390429e-03 ||r(i)||/||b|| 1.587586956323e-02 3 KSP unpreconditioned resid norm 1.115554835302e-03 true resid norm 1.115554835302e-03 ||r(i)||/||b|| 1.587574781302e-02 4 KSP unpreconditioned resid norm 1.115554764006e-03 true resid norm 1.115554764006e-03 ||r(i)||/||b|| 1.587574679839e-02 5 KSP unpreconditioned resid norm 1.115548853894e-03 true resid norm 1.115548853906e-03 ||r(i)||/||b|| 1.587566269024e-02 6 KSP unpreconditioned resid norm 7.632184921082e-05 true resid norm 7.632265261003e-05 ||r(i)||/||b|| 1.086167301610e-03 7 KSP unpreconditioned resid norm 1.422983940334e-08 true resid norm 4.141122043271e-08 ||r(i)||/||b|| 5.893337300996e-07 2 SNES Function norm 8.778069005998e-06 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 8.778069005998e-06 true resid norm 8.778069005998e-06 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 3.525945734318e-06 true resid norm 3.525945734318e-06 ||r(i)||/||b|| 4.016766935768e-01 2 KSP unpreconditioned resid norm 7.092855329067e-07 true resid norm 7.092855329067e-07 ||r(i)||/||b|| 8.080200012349e-02 3 KSP unpreconditioned resid norm 7.092827495949e-07 true resid norm 7.092827495949e-07 ||r(i)||/||b|| 8.080168304786e-02 4 KSP unpreconditioned resid norm 7.092826142571e-07 true resid norm 7.092826142574e-07 ||r(i)||/||b|| 8.080166763017e-02 5 KSP unpreconditioned resid norm 8.490734432004e-08 true resid norm 8.490728815768e-08 ||r(i)||/||b|| 9.672661276604e-03 6 KSP unpreconditioned resid norm 3.681976478361e-12 true resid norm 9.310211086004e-12 ||r(i)||/||b|| 1.060621769963e-06 3 SNES Function norm 9.754530371801e-12 N = 10 0 SNES Function norm 3.857209122044e+00 1 SNES Function norm 2.174942016837e-02 2 SNES Function norm 7.117856370756e-04 3 SNES Function norm 7.102760546744e-04 4 SNES Function norm 7.102760536032e-04 0 SNES Function norm 3.857209122044e+00 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 3.857209122044e+00 true resid norm 3.857209122044e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.386167872908e-01 true resid norm 1.386167872908e-01 ||r(i)||/||b|| 3.593706820265e-02 2 KSP unpreconditioned resid norm 4.475586255809e-02 true resid norm 4.475586255809e-02 ||r(i)||/||b|| 1.160317243426e-02 3 KSP unpreconditioned resid norm 1.722695448041e-02 true resid norm 1.722695448041e-02 ||r(i)||/||b|| 4.466170730012e-03 4 KSP unpreconditioned resid norm 5.231768306662e-03 true resid norm 5.231768306662e-03 ||r(i)||/||b|| 1.356361073804e-03 5 KSP unpreconditioned resid norm 2.041434254281e-03 true resid norm 2.041434254281e-03 ||r(i)||/||b|| 5.292516401597e-04 6 KSP unpreconditioned resid norm 5.875484274903e-04 true resid norm 5.875484274903e-04 ||r(i)||/||b|| 1.523247531829e-04 7 KSP unpreconditioned resid norm 2.354394338936e-04 true resid norm 2.354394338936e-04 ||r(i)||/||b|| 6.103880459787e-05 8 KSP unpreconditioned resid norm 1.414423189101e-04 true resid norm 1.414423189101e-04 ||r(i)||/||b|| 3.666960085254e-05 9 KSP unpreconditioned resid norm 1.272526161323e-04 true resid norm 1.272526161323e-04 ||r(i)||/||b|| 3.299085222139e-05 10 KSP unpreconditioned resid norm 1.256180185176e-04 true resid norm 1.256180185176e-04 ||r(i)||/||b|| 3.256707493500e-05 11 KSP unpreconditioned resid norm 1.253360959842e-04 true resid norm 1.253360959842e-04 ||r(i)||/||b|| 3.249398516348e-05 12 KSP unpreconditioned resid norm 1.252954886770e-04 true resid norm 1.252954886770e-04 ||r(i)||/||b|| 3.248345752397e-05 13 KSP unpreconditioned resid norm 1.252778828690e-04 true resid norm 1.252778828690e-04 ||r(i)||/||b|| 3.247889313364e-05 14 KSP unpreconditioned resid norm 1.252729271101e-04 true resid norm 1.252729271101e-04 ||r(i)||/||b|| 3.247760832934e-05 15 KSP unpreconditioned resid norm 1.252709541672e-04 true resid norm 1.252709541672e-04 ||r(i)||/||b|| 3.247709683442e-05 16 KSP unpreconditioned resid norm 1.252698774325e-04 true resid norm 1.252698774325e-04 ||r(i)||/||b|| 3.247681768576e-05 17 KSP unpreconditioned resid norm 1.252693026661e-04 true resid norm 1.252693026661e-04 ||r(i)||/||b|| 3.247666867482e-05 18 KSP unpreconditioned resid norm 1.252691813938e-04 true resid norm 1.252691813938e-04 ||r(i)||/||b|| 3.247663723438e-05 19 KSP unpreconditioned resid norm 1.252690631661e-04 true resid norm 1.252690631661e-04 ||r(i)||/||b|| 3.247660658329e-05 20 KSP unpreconditioned resid norm 1.252682040589e-04 true resid norm 1.252682040589e-04 ||r(i)||/||b|| 3.247638385562e-05 21 KSP unpreconditioned resid norm 1.252666235238e-04 true resid norm 1.252666235237e-04 ||r(i)||/||b|| 3.247597409427e-05 22 KSP unpreconditioned resid norm 1.252663551040e-04 true resid norm 1.252663551038e-04 ||r(i)||/||b|| 3.247590450512e-05 23 KSP unpreconditioned resid norm 1.252657099958e-04 true resid norm 1.252657099961e-04 ||r(i)||/||b|| 3.247573725784e-05 24 KSP unpreconditioned resid norm 1.252222519823e-04 true resid norm 1.252222519982e-04 ||r(i)||/||b|| 3.246447056306e-05 25 KSP unpreconditioned resid norm 1.248107616721e-04 true resid norm 1.248107618126e-04 ||r(i)||/||b|| 3.235778975510e-05 26 KSP unpreconditioned resid norm 1.197171374954e-04 true resid norm 1.197171390035e-04 ||r(i)||/||b|| 3.103724356537e-05 27 KSP unpreconditioned resid norm 9.324077689908e-05 true resid norm 9.324078248863e-05 ||r(i)||/||b|| 2.417312091163e-05 28 KSP unpreconditioned resid norm 3.902477473206e-05 true resid norm 3.902477892599e-05 ||r(i)||/||b|| 1.011736146297e-05 29 KSP unpreconditioned resid norm 1.307801475003e-05 true resid norm 1.307801741838e-05 ||r(i)||/||b|| 3.390538859724e-06 1 SNES Function norm 2.171186815034e-02 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 2.171186815034e-02 true resid norm 2.171186815034e-02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 5.755446049476e-03 true resid norm 5.755446049476e-03 ||r(i)||/||b|| 2.650829495474e-01 2 KSP unpreconditioned resid norm 1.819729526723e-03 true resid norm 1.819729526723e-03 ||r(i)||/||b|| 8.381266476578e-02 3 KSP unpreconditioned resid norm 5.868270016474e-04 true resid norm 5.868270016474e-04 ||r(i)||/||b|| 2.702793686771e-02 4 KSP unpreconditioned resid norm 1.891824694427e-04 true resid norm 1.891824694427e-04 ||r(i)||/||b|| 8.713320665581e-03 5 KSP unpreconditioned resid norm 7.091162609666e-05 true resid norm 7.091162609666e-05 ||r(i)||/||b|| 3.266030615406e-03 6 KSP unpreconditioned resid norm 5.637701208830e-05 true resid norm 5.637701208830e-05 ||r(i)||/||b|| 2.596598860030e-03 7 KSP unpreconditioned resid norm 5.383999628854e-05 true resid norm 5.383999628854e-05 ||r(i)||/||b|| 2.479749596660e-03 8 KSP unpreconditioned resid norm 5.369249277053e-05 true resid norm 5.369249277053e-05 ||r(i)||/||b|| 2.472955915113e-03 9 KSP unpreconditioned resid norm 5.368343313838e-05 true resid norm 5.368343313838e-05 ||r(i)||/||b|| 2.472538648755e-03 10 KSP unpreconditioned resid norm 5.368324507663e-05 true resid norm 5.368324507663e-05 ||r(i)||/||b|| 2.472529987052e-03 11 KSP unpreconditioned resid norm 5.367688316238e-05 true resid norm 5.367688316238e-05 ||r(i)||/||b|| 2.472236971536e-03 12 KSP unpreconditioned resid norm 5.364660764375e-05 true resid norm 5.364660764375e-05 ||r(i)||/||b|| 2.470842548982e-03 13 KSP unpreconditioned resid norm 5.363778445858e-05 true resid norm 5.363778445858e-05 ||r(i)||/||b|| 2.470436172843e-03 14 KSP unpreconditioned resid norm 5.363146760267e-05 true resid norm 5.363146760267e-05 ||r(i)||/||b|| 2.470145232613e-03 15 KSP unpreconditioned resid norm 5.363036199206e-05 true resid norm 5.363036199206e-05 ||r(i)||/||b|| 2.470094310665e-03 16 KSP unpreconditioned resid norm 5.362586041467e-05 true resid norm 5.362586041467e-05 ||r(i)||/||b|| 2.469886978097e-03 17 KSP unpreconditioned resid norm 5.362298140611e-05 true resid norm 5.362298140610e-05 ||r(i)||/||b|| 2.469754377413e-03 18 KSP unpreconditioned resid norm 5.362041508854e-05 true resid norm 5.362041508851e-05 ||r(i)||/||b|| 2.469636178574e-03 19 KSP unpreconditioned resid norm 5.362023682045e-05 true resid norm 5.362023682040e-05 ||r(i)||/||b|| 2.469627967945e-03 20 KSP unpreconditioned resid norm 5.362021431482e-05 true resid norm 5.362021431475e-05 ||r(i)||/||b|| 2.469626931385e-03 21 KSP unpreconditioned resid norm 5.362021307632e-05 true resid norm 5.362021307624e-05 ||r(i)||/||b|| 2.469626874342e-03 22 KSP unpreconditioned resid norm 5.362020945699e-05 true resid norm 5.362020945697e-05 ||r(i)||/||b|| 2.469626707646e-03 23 KSP unpreconditioned resid norm 5.360919215998e-05 true resid norm 5.360919217038e-05 ||r(i)||/||b|| 2.469119276111e-03 24 KSP unpreconditioned resid norm 5.352778280091e-05 true resid norm 5.352778288656e-05 ||r(i)||/||b|| 2.465369746902e-03 25 KSP unpreconditioned resid norm 5.256642319528e-05 true resid norm 5.256642407158e-05 ||r(i)||/||b|| 2.421091713877e-03 26 KSP unpreconditioned resid norm 4.729160583219e-05 true resid norm 4.729161069155e-05 ||r(i)||/||b|| 2.178145628192e-03 27 KSP unpreconditioned resid norm 2.668806077499e-05 true resid norm 2.668806954502e-05 ||r(i)||/||b|| 1.229192686701e-03 28 KSP unpreconditioned resid norm 1.020558332703e-05 true resid norm 1.020558761616e-05 ||r(i)||/||b|| 4.700464992463e-04 29 KSP unpreconditioned resid norm 3.451738103984e-06 true resid norm 3.451741888987e-06 ||r(i)||/||b|| 1.589794975304e-04 30 KSP unpreconditioned resid norm 1.120801314646e-06 true resid norm 1.120801314646e-06 ||r(i)||/||b|| 5.162159731651e-05 31 KSP unpreconditioned resid norm 6.657758394387e-07 true resid norm 6.657758394387e-07 ||r(i)||/||b|| 3.066414344582e-05 32 KSP unpreconditioned resid norm 1.791100894652e-07 true resid norm 1.791100894652e-07 ||r(i)||/||b|| 8.249409411708e-06 2 SNES Function norm 1.208714436027e-06 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 1.208714436027e-06 true resid norm 1.208714436027e-06 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 7.009704990544e-07 true resid norm 7.009704990544e-07 ||r(i)||/||b|| 5.799306090514e-01 2 KSP unpreconditioned resid norm 2.928942580645e-07 true resid norm 2.928942580645e-07 ||r(i)||/||b|| 2.423188218279e-01 3 KSP unpreconditioned resid norm 1.900700846501e-07 true resid norm 1.900700846501e-07 ||r(i)||/||b|| 1.572497845520e-01 4 KSP unpreconditioned resid norm 1.694415211872e-07 true resid norm 1.694415211872e-07 ||r(i)||/||b|| 1.401832526665e-01 5 KSP unpreconditioned resid norm 1.670108100598e-07 true resid norm 1.670108100598e-07 ||r(i)||/||b|| 1.381722639210e-01 6 KSP unpreconditioned resid norm 1.668707304029e-07 true resid norm 1.668707304029e-07 ||r(i)||/||b|| 1.380563724807e-01 7 KSP unpreconditioned resid norm 1.668026114154e-07 true resid norm 1.668026114154e-07 ||r(i)||/||b|| 1.380000159208e-01 8 KSP unpreconditioned resid norm 1.668025617944e-07 true resid norm 1.668025617944e-07 ||r(i)||/||b|| 1.379999748681e-01 9 KSP unpreconditioned resid norm 1.667959318665e-07 true resid norm 1.667959318665e-07 ||r(i)||/||b|| 1.379944897612e-01 10 KSP unpreconditioned resid norm 1.667495881623e-07 true resid norm 1.667495881623e-07 ||r(i)||/||b|| 1.379561484435e-01 11 KSP unpreconditioned resid norm 1.667472366297e-07 true resid norm 1.667472366297e-07 ||r(i)||/||b|| 1.379542029611e-01 12 KSP unpreconditioned resid norm 1.667421748942e-07 true resid norm 1.667421748942e-07 ||r(i)||/||b|| 1.379500152595e-01 13 KSP unpreconditioned resid norm 1.667405719095e-07 true resid norm 1.667405719095e-07 ||r(i)||/||b|| 1.379486890697e-01 14 KSP unpreconditioned resid norm 1.667313112753e-07 true resid norm 1.667313112754e-07 ||r(i)||/||b|| 1.379410275130e-01 15 KSP unpreconditioned resid norm 1.667263791233e-07 true resid norm 1.667263791233e-07 ||r(i)||/||b|| 1.379369470190e-01 16 KSP unpreconditioned resid norm 1.666960300587e-07 true resid norm 1.666960300590e-07 ||r(i)||/||b|| 1.379118384711e-01 17 KSP unpreconditioned resid norm 1.666924323014e-07 true resid norm 1.666924323014e-07 ||r(i)||/||b|| 1.379088619553e-01 18 KSP unpreconditioned resid norm 1.666835606480e-07 true resid norm 1.666835606492e-07 ||r(i)||/||b|| 1.379015222132e-01 19 KSP unpreconditioned resid norm 1.666784002237e-07 true resid norm 1.666784002274e-07 ||r(i)||/||b|| 1.378972528659e-01 20 KSP unpreconditioned resid norm 1.665390347333e-07 true resid norm 1.665390347759e-07 ||r(i)||/||b|| 1.377819523057e-01 21 KSP unpreconditioned resid norm 1.662598119876e-07 true resid norm 1.662598121924e-07 ||r(i)||/||b|| 1.375509444058e-01 22 KSP unpreconditioned resid norm 1.625064404254e-07 true resid norm 1.625064423607e-07 ||r(i)||/||b|| 1.344456866875e-01 23 KSP unpreconditioned resid norm 1.391707373331e-07 true resid norm 1.391707472513e-07 ||r(i)||/||b|| 1.151394763752e-01 24 KSP unpreconditioned resid norm 7.505751761619e-08 true resid norm 7.505754230672e-08 ||r(i)||/||b|| 6.209700163211e-02 25 KSP unpreconditioned resid norm 2.799964315999e-08 true resid norm 2.799965549375e-08 ||r(i)||/||b|| 2.316482260755e-02 26 KSP unpreconditioned resid norm 9.970551878855e-09 true resid norm 9.970561618435e-09 ||r(i)||/||b|| 8.248897606624e-03 27 KSP unpreconditioned resid norm 3.252551635867e-09 true resid norm 3.252558544160e-09 ||r(i)||/||b|| 2.690923883437e-03 28 KSP unpreconditioned resid norm 1.136728449942e-09 true resid norm 1.136739747171e-09 ||r(i)||/||b|| 9.404535209387e-04 29 KSP unpreconditioned resid norm 3.672093573536e-10 true resid norm 3.672155322102e-10 ||r(i)||/||b|| 3.038066902032e-04 30 KSP unpreconditioned resid norm 1.234864719987e-10 true resid norm 1.234864719987e-10 ||r(i)||/||b|| 1.021634790800e-04 31 KSP unpreconditioned resid norm 6.510725158420e-11 true resid norm 6.510725158421e-11 ||r(i)||/||b|| 5.386487464998e-05 32 KSP unpreconditioned resid norm 1.805042777062e-11 true resid norm 1.805042777059e-11 ||r(i)||/||b|| 1.493357507164e-05 33 KSP unpreconditioned resid norm 7.492522181774e-12 true resid norm 7.492522181758e-12 ||r(i)||/||b|| 6.198752954739e-06 3 SNES Function norm 7.512220214011e-12 N=20 0 SNES Function norm 3.857209122044e+00 1 SNES Function norm 2.174942016837e-02 2 SNES Function norm 7.117856370756e-04 3 SNES Function norm 7.102760546744e-04 4 SNES Function norm 7.102760536032e-04 0 SNES Function norm 3.857209122044e+00 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 3.857209122044e+00 true resid norm 3.857209122044e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.386167872908e-01 true resid norm 1.386167872908e-01 ||r(i)||/||b|| 3.593706820265e-02 2 KSP unpreconditioned resid norm 4.475586255809e-02 true resid norm 4.475586255809e-02 ||r(i)||/||b|| 1.160317243426e-02 3 KSP unpreconditioned resid norm 1.722695448041e-02 true resid norm 1.722695448041e-02 ||r(i)||/||b|| 4.466170730012e-03 4 KSP unpreconditioned resid norm 5.231768306662e-03 true resid norm 5.231768306662e-03 ||r(i)||/||b|| 1.356361073804e-03 5 KSP unpreconditioned resid norm 2.041434254281e-03 true resid norm 2.041434254281e-03 ||r(i)||/||b|| 5.292516401597e-04 6 KSP unpreconditioned resid norm 5.875484274903e-04 true resid norm 5.875484274903e-04 ||r(i)||/||b|| 1.523247531829e-04 7 KSP unpreconditioned resid norm 2.354394338936e-04 true resid norm 2.354394338936e-04 ||r(i)||/||b|| 6.103880459787e-05 8 KSP unpreconditioned resid norm 1.414423189101e-04 true resid norm 1.414423189101e-04 ||r(i)||/||b|| 3.666960085254e-05 9 KSP unpreconditioned resid norm 1.272526161323e-04 true resid norm 1.272526161323e-04 ||r(i)||/||b|| 3.299085222139e-05 10 KSP unpreconditioned resid norm 1.256180185176e-04 true resid norm 1.256180185176e-04 ||r(i)||/||b|| 3.256707493500e-05 11 KSP unpreconditioned resid norm 1.253360959842e-04 true resid norm 1.253360959842e-04 ||r(i)||/||b|| 3.249398516348e-05 12 KSP unpreconditioned resid norm 1.252954886770e-04 true resid norm 1.252954886770e-04 ||r(i)||/||b|| 3.248345752397e-05 13 KSP unpreconditioned resid norm 1.252778828690e-04 true resid norm 1.252778828690e-04 ||r(i)||/||b|| 3.247889313364e-05 14 KSP unpreconditioned resid norm 1.252729271101e-04 true resid norm 1.252729271101e-04 ||r(i)||/||b|| 3.247760832934e-05 15 KSP unpreconditioned resid norm 1.252709541672e-04 true resid norm 1.252709541672e-04 ||r(i)||/||b|| 3.247709683442e-05 16 KSP unpreconditioned resid norm 1.252698774325e-04 true resid norm 1.252698774325e-04 ||r(i)||/||b|| 3.247681768576e-05 17 KSP unpreconditioned resid norm 1.252693026661e-04 true resid norm 1.252693026661e-04 ||r(i)||/||b|| 3.247666867482e-05 18 KSP unpreconditioned resid norm 1.252691813938e-04 true resid norm 1.252691813938e-04 ||r(i)||/||b|| 3.247663723438e-05 19 KSP unpreconditioned resid norm 1.252690631661e-04 true resid norm 1.252690631661e-04 ||r(i)||/||b|| 3.247660658329e-05 20 KSP unpreconditioned resid norm 1.252682040589e-04 true resid norm 1.252682040589e-04 ||r(i)||/||b|| 3.247638385562e-05 21 KSP unpreconditioned resid norm 1.252666235238e-04 true resid norm 1.252666235237e-04 ||r(i)||/||b|| 3.247597409427e-05 22 KSP unpreconditioned resid norm 1.252663551040e-04 true resid norm 1.252663551038e-04 ||r(i)||/||b|| 3.247590450512e-05 23 KSP unpreconditioned resid norm 1.252657099958e-04 true resid norm 1.252657099961e-04 ||r(i)||/||b|| 3.247573725784e-05 24 KSP unpreconditioned resid norm 1.252222519823e-04 true resid norm 1.252222519982e-04 ||r(i)||/||b|| 3.246447056306e-05 25 KSP unpreconditioned resid norm 1.248107616721e-04 true resid norm 1.248107618126e-04 ||r(i)||/||b|| 3.235778975510e-05 26 KSP unpreconditioned resid norm 1.197171374954e-04 true resid norm 1.197171390035e-04 ||r(i)||/||b|| 3.103724356537e-05 27 KSP unpreconditioned resid norm 9.324077689908e-05 true resid norm 9.324078248863e-05 ||r(i)||/||b|| 2.417312091163e-05 28 KSP unpreconditioned resid norm 3.902477473206e-05 true resid norm 3.902477892599e-05 ||r(i)||/||b|| 1.011736146297e-05 29 KSP unpreconditioned resid norm 1.307801475003e-05 true resid norm 1.307801741838e-05 ||r(i)||/||b|| 3.390538859724e-06 1 SNES Function norm 2.171186815034e-02 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 2.171186815034e-02 true resid norm 2.171186815034e-02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 5.755446049476e-03 true resid norm 5.755446049476e-03 ||r(i)||/||b|| 2.650829495474e-01 2 KSP unpreconditioned resid norm 1.819729526723e-03 true resid norm 1.819729526723e-03 ||r(i)||/||b|| 8.381266476578e-02 3 KSP unpreconditioned resid norm 5.868270016474e-04 true resid norm 5.868270016474e-04 ||r(i)||/||b|| 2.702793686771e-02 4 KSP unpreconditioned resid norm 1.891824694427e-04 true resid norm 1.891824694427e-04 ||r(i)||/||b|| 8.713320665581e-03 5 KSP unpreconditioned resid norm 7.091162609666e-05 true resid norm 7.091162609666e-05 ||r(i)||/||b|| 3.266030615406e-03 6 KSP unpreconditioned resid norm 5.637701208830e-05 true resid norm 5.637701208830e-05 ||r(i)||/||b|| 2.596598860030e-03 7 KSP unpreconditioned resid norm 5.383999628854e-05 true resid norm 5.383999628854e-05 ||r(i)||/||b|| 2.479749596660e-03 8 KSP unpreconditioned resid norm 5.369249277053e-05 true resid norm 5.369249277053e-05 ||r(i)||/||b|| 2.472955915113e-03 9 KSP unpreconditioned resid norm 5.368343313838e-05 true resid norm 5.368343313838e-05 ||r(i)||/||b|| 2.472538648755e-03 10 KSP unpreconditioned resid norm 5.368324507663e-05 true resid norm 5.368324507663e-05 ||r(i)||/||b|| 2.472529987052e-03 11 KSP unpreconditioned resid norm 5.367688316238e-05 true resid norm 5.367688316238e-05 ||r(i)||/||b|| 2.472236971536e-03 12 KSP unpreconditioned resid norm 5.364660764375e-05 true resid norm 5.364660764375e-05 ||r(i)||/||b|| 2.470842548982e-03 13 KSP unpreconditioned resid norm 5.363778445858e-05 true resid norm 5.363778445858e-05 ||r(i)||/||b|| 2.470436172843e-03 14 KSP unpreconditioned resid norm 5.363146760267e-05 true resid norm 5.363146760267e-05 ||r(i)||/||b|| 2.470145232613e-03 15 KSP unpreconditioned resid norm 5.363036199206e-05 true resid norm 5.363036199206e-05 ||r(i)||/||b|| 2.470094310665e-03 16 KSP unpreconditioned resid norm 5.362586041467e-05 true resid norm 5.362586041467e-05 ||r(i)||/||b|| 2.469886978097e-03 17 KSP unpreconditioned resid norm 5.362298140611e-05 true resid norm 5.362298140610e-05 ||r(i)||/||b|| 2.469754377413e-03 18 KSP unpreconditioned resid norm 5.362041508854e-05 true resid norm 5.362041508851e-05 ||r(i)||/||b|| 2.469636178574e-03 19 KSP unpreconditioned resid norm 5.362023682045e-05 true resid norm 5.362023682040e-05 ||r(i)||/||b|| 2.469627967945e-03 20 KSP unpreconditioned resid norm 5.362021431482e-05 true resid norm 5.362021431475e-05 ||r(i)||/||b|| 2.469626931385e-03 21 KSP unpreconditioned resid norm 5.362021307632e-05 true resid norm 5.362021307624e-05 ||r(i)||/||b|| 2.469626874342e-03 22 KSP unpreconditioned resid norm 5.362020945699e-05 true resid norm 5.362020945697e-05 ||r(i)||/||b|| 2.469626707646e-03 23 KSP unpreconditioned resid norm 5.360919215998e-05 true resid norm 5.360919217038e-05 ||r(i)||/||b|| 2.469119276111e-03 24 KSP unpreconditioned resid norm 5.352778280091e-05 true resid norm 5.352778288656e-05 ||r(i)||/||b|| 2.465369746902e-03 25 KSP unpreconditioned resid norm 5.256642319528e-05 true resid norm 5.256642407158e-05 ||r(i)||/||b|| 2.421091713877e-03 26 KSP unpreconditioned resid norm 4.729160583219e-05 true resid norm 4.729161069155e-05 ||r(i)||/||b|| 2.178145628192e-03 27 KSP unpreconditioned resid norm 2.668806077499e-05 true resid norm 2.668806954502e-05 ||r(i)||/||b|| 1.229192686701e-03 28 KSP unpreconditioned resid norm 1.020558332703e-05 true resid norm 1.020558761616e-05 ||r(i)||/||b|| 4.700464992463e-04 29 KSP unpreconditioned resid norm 3.451738103984e-06 true resid norm 3.451741888987e-06 ||r(i)||/||b|| 1.589794975304e-04 30 KSP unpreconditioned resid norm 1.120801314646e-06 true resid norm 1.120801314646e-06 ||r(i)||/||b|| 5.162159731651e-05 31 KSP unpreconditioned resid norm 6.657758394387e-07 true resid norm 6.657758394387e-07 ||r(i)||/||b|| 3.066414344582e-05 32 KSP unpreconditioned resid norm 1.791100894652e-07 true resid norm 1.791100894652e-07 ||r(i)||/||b|| 8.249409411708e-06 2 SNES Function norm 1.208714436027e-06 Residual norms for firedrake_1_ solve. 0 KSP unpreconditioned resid norm 1.208714436027e-06 true resid norm 1.208714436027e-06 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 7.009704990544e-07 true resid norm 7.009704990544e-07 ||r(i)||/||b|| 5.799306090514e-01 2 KSP unpreconditioned resid norm 2.928942580645e-07 true resid norm 2.928942580645e-07 ||r(i)||/||b|| 2.423188218279e-01 3 KSP unpreconditioned resid norm 1.900700846501e-07 true resid norm 1.900700846501e-07 ||r(i)||/||b|| 1.572497845520e-01 4 KSP unpreconditioned resid norm 1.694415211872e-07 true resid norm 1.694415211872e-07 ||r(i)||/||b|| 1.401832526665e-01 5 KSP unpreconditioned resid norm 1.670108100598e-07 true resid norm 1.670108100598e-07 ||r(i)||/||b|| 1.381722639210e-01 6 KSP unpreconditioned resid norm 1.668707304029e-07 true resid norm 1.668707304029e-07 ||r(i)||/||b|| 1.380563724807e-01 7 KSP unpreconditioned resid norm 1.668026114154e-07 true resid norm 1.668026114154e-07 ||r(i)||/||b|| 1.380000159208e-01 8 KSP unpreconditioned resid norm 1.668025617944e-07 true resid norm 1.668025617944e-07 ||r(i)||/||b|| 1.379999748681e-01 9 KSP unpreconditioned resid norm 1.667959318665e-07 true resid norm 1.667959318665e-07 ||r(i)||/||b|| 1.379944897612e-01 10 KSP unpreconditioned resid norm 1.667495881623e-07 true resid norm 1.667495881623e-07 ||r(i)||/||b|| 1.379561484435e-01 11 KSP unpreconditioned resid norm 1.667472366297e-07 true resid norm 1.667472366297e-07 ||r(i)||/||b|| 1.379542029611e-01 12 KSP unpreconditioned resid norm 1.667421748942e-07 true resid norm 1.667421748942e-07 ||r(i)||/||b|| 1.379500152595e-01 13 KSP unpreconditioned resid norm 1.667405719095e-07 true resid norm 1.667405719095e-07 ||r(i)||/||b|| 1.379486890697e-01 14 KSP unpreconditioned resid norm 1.667313112753e-07 true resid norm 1.667313112754e-07 ||r(i)||/||b|| 1.379410275130e-01 15 KSP unpreconditioned resid norm 1.667263791233e-07 true resid norm 1.667263791233e-07 ||r(i)||/||b|| 1.379369470190e-01 16 KSP unpreconditioned resid norm 1.666960300587e-07 true resid norm 1.666960300590e-07 ||r(i)||/||b|| 1.379118384711e-01 17 KSP unpreconditioned resid norm 1.666924323014e-07 true resid norm 1.666924323014e-07 ||r(i)||/||b|| 1.379088619553e-01 18 KSP unpreconditioned resid norm 1.666835606480e-07 true resid norm 1.666835606492e-07 ||r(i)||/||b|| 1.379015222132e-01 19 KSP unpreconditioned resid norm 1.666784002237e-07 true resid norm 1.666784002274e-07 ||r(i)||/||b|| 1.378972528659e-01 20 KSP unpreconditioned resid norm 1.665390347333e-07 true resid norm 1.665390347759e-07 ||r(i)||/||b|| 1.377819523057e-01 21 KSP unpreconditioned resid norm 1.662598119876e-07 true resid norm 1.662598121924e-07 ||r(i)||/||b|| 1.375509444058e-01 22 KSP unpreconditioned resid norm 1.625064404254e-07 true resid norm 1.625064423607e-07 ||r(i)||/||b|| 1.344456866875e-01 23 KSP unpreconditioned resid norm 1.391707373331e-07 true resid norm 1.391707472513e-07 ||r(i)||/||b|| 1.151394763752e-01 24 KSP unpreconditioned resid norm 7.505751761619e-08 true resid norm 7.505754230672e-08 ||r(i)||/||b|| 6.209700163211e-02 25 KSP unpreconditioned resid norm 2.799964315999e-08 true resid norm 2.799965549375e-08 ||r(i)||/||b|| 2.316482260755e-02 26 KSP unpreconditioned resid norm 9.970551878855e-09 true resid norm 9.970561618435e-09 ||r(i)||/||b|| 8.248897606624e-03 27 KSP unpreconditioned resid norm 3.252551635867e-09 true resid norm 3.252558544160e-09 ||r(i)||/||b|| 2.690923883437e-03 28 KSP unpreconditioned resid norm 1.136728449942e-09 true resid norm 1.136739747171e-09 ||r(i)||/||b|| 9.404535209387e-04 29 KSP unpreconditioned resid norm 3.672093573536e-10 true resid norm 3.672155322102e-10 ||r(i)||/||b|| 3.038066902032e-04 30 KSP unpreconditioned resid norm 1.234864719987e-10 true resid norm 1.234864719987e-10 ||r(i)||/||b|| 1.021634790800e-04 31 KSP unpreconditioned resid norm 6.510725158420e-11 true resid norm 6.510725158421e-11 ||r(i)||/||b|| 5.386487464998e-05 32 KSP unpreconditioned resid norm 1.805042777062e-11 true resid norm 1.805042777059e-11 ||r(i)||/||b|| 1.493357507164e-05 33 KSP unpreconditioned resid norm 7.492522181774e-12 true resid norm 7.492522181758e-12 ||r(i)||/||b|| 6.198752954739e-06 3 SNES Function norm 7.512220214011e-12 Best, Zizhou On Tue, Apr 13, 2021 at 10:04 AM Matthew Knepley <knepley@gmail.com<mailto:knepley@gmail.com>> wrote: On Mon, Apr 12, 2021 at 9:45 PM Zizhou Huang <zizhou@nyu.edu<mailto:zizhou@nyu.edu>> wrote: Hi Matthew, Thanks for your advice! I did some basic tests with the Navier-Stokes solver inside the Firedrake. For the lid-driven cavity problem on a NxNxN grid, it takes 23sec for N=5, 594sec for N=10, and 7424sec for N=20. For N=40 it almost takes forever, but it's really not a dense mesh in practice. I had the same experience with my own implementation of fem, using Pardiso to solve the linear system directly, and I've been finding efficient linear solvers to get rid of this problem, so I'm asking if the Firedrake team has any better options for the linear solver. What solver did you use and how many iterates did it take? Also, posting the logs makes things easier. Thanks Matt Best, Zizhou On Mon, Apr 12, 2021 at 11:21 PM Matthew Knepley <knepley@gmail.com<mailto:knepley@gmail.com>> wrote: On Mon, Apr 12, 2021 at 11:00 AM Zizhou Huang <zizhou@nyu.edu<mailto:zizhou@nyu.edu>> wrote: To one whom it may concern, I'm a PhD student at NYU. I'm working on a project on solving the Navier-Stokes equation with FEM, so I'm really interested in the amazing demo of solving the Navier-Stokes equation on the website. (https://www.firedrakeproject.org/demos/navier_stokes.py.html) I modified the code to solve the 3d lid-driven cavity problem with a uniform grid of size 40x40x40, and found that it takes extremely long time and large memory. I had the same issue with my code of similar sizes, which uses Pardiso to solve the linear system. I'm wondering if fem is indeed unsolvable for a large problem like this? Do you have any experience to make it faster by changing the parameters in the linear solver? When you want to understand how a solver is scaling, start with a small problem, and look at the performance with -log_view. Then start increasing the problem size, and look at how the number of iterates, subiterates, and event times scale. Once you have that data, getting help is much easier. Thanks, Matt Looking forward to hearing from you. Best, Zizhou _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk<mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

On 13 Apr 2021, at 14:15, Zizhou Huang <zizhou@nyu.edu> wrote:

Thanks for your reply! Do you have any idea of which inner solver can do a much better job? I tried to replace lu with hypre, but it's still very slow for large 3d problems, like solving on a 40x40x40 grid.

It depends a bit on what your Reynolds number is: what kind of regimes do you want to handle? I'd encourage you to consider the augmented Lagrangian approach that Patrick described in his message yesterday, we even have a Firedrake-based solver you can install and use at https://github.com/florianwechsung/alfi. This approach is robust in the Reynolds number (up to 10000 for stationary, and larger for time-dependent flow), and scales to large problems as Patrick mentioned. Apart from that here's some more generic advice on debugging "slowness" of solver convergence for block preconditioners. With these block preconditioners "slowness" can come from many places. It might be that just applying the inner preconditioners takes a long time (e.g. LU factorisation in 3D), or, the iterative method you're using on the inside is not a good approximation to the inverse and so the inner solves need many iterations. Finally, the Schur complement approximation may not be good, so you need many outer iterations to clean things up. To understand which is going on, we'd need to see the output including logs for the outer KSP "ksp_monitor_true_residual": None And the inner KSPs "fieldsplit_0_ksp_monitor_true_residual": None "fieldsplit_1_ksp_monitor_true_residual": None In terms of choices, for the top-left block (i.e. the momentum equation) then the best algebraic multigrid methods are based on Approximate Ideal Restriction (e.g. https://arxiv.org/abs/2010.11130). An AIR-based AMG is implemented in MueLu, but I think that is not exposed from PETSc. Alternately, you could use the smoothed aggregation version of ML (-fieldsplit_0_pc_type ml), which is what Patrick, Florian, and I did in https://arxiv.org/pdf/1810.03315.pdf (section 5.4.1). For the scalar Laplacian inverse in the Schur complement approximation, Hypre or ML should work well, but note that the PCD approximation degrades approximately with the square root of the Reynolds number (so above perhaps Re=100 it starts to struggle). I also note that the PCD implementation in Firedrake doesn't correctly handle all types of boundary conditions (e.g. it is sub-optimal for inflow or outflow conditions). The correct treatment is described in Section 3 of Jan Blechta's thesis (https://dspace.cuni.cz/bitstream/handle/20.500.11956/108384/140075745.pdf) in particular the summary in 3.5 is good. We can provide advice on how to do this in the existing PCD framework. Lawrence

On Tue, Apr 13, 2021 at 9:37 AM Lawrence Mitchell <wence@gmx.li> wrote:

...

On 13 Apr 2021, at 14:15, Zizhou Huang <zizhou@nyu.edu> wrote:

Thanks for your reply! Do you have any idea of which inner solver can do a much better job? I tried to replace lu with hypre, but it's still very slow for large 3d problems, like solving on a 40x40x40 grid.

It depends a bit on what your Reynolds number is: what kind of regimes do you want to handle? I'd encourage you to consider the augmented Lagrangian approach that Patrick described in his message yesterday, we even have a Firedrake-based solver you can install and use at https://github.com/florianwechsung/alfi. This approach is robust in the Reynolds number (up to 10000 for stationary, and larger for time-dependent flow), and scales to large problems as Patrick mentioned.

Apart from that here's some more generic advice on debugging "slowness" of solver convergence for block preconditioners.

With these block preconditioners "slowness" can come from many places. It might be that just applying the inner preconditioners takes a long time (e.g. LU factorisation in 3D), or, the iterative method you're using on the inside is not a good approximation to the inverse and so the inner solves need many iterations. Finally, the Schur complement approximation may not be good, so you need many outer iterations to clean things up. To understand which is going on, we'd need to see the output including logs for the outer KSP

"ksp_monitor_true_residual": None

And the inner KSPs

"fieldsplit_0_ksp_monitor_true_residual": None "fieldsplit_1_ksp_monitor_true_residual": None

In terms of choices, for the top-left block (i.e. the momentum equation) then the best algebraic multigrid methods are based on Approximate Ideal Restriction (e.g. https://arxiv.org/abs/2010.11130).

I think this overreaches slightly, AIR is good when the diffusion is very low, but in the middle regime and also diffusion limited it can be quite poor. Thanks, Matt

An AIR-based AMG is implemented in MueLu, but I think that is not exposed from PETSc. Alternately, you could use the smoothed aggregation version of ML (-fieldsplit_0_pc_type ml), which is what Patrick, Florian, and I did in https://arxiv.org/pdf/1810.03315.pdf (section 5.4.1).

For the scalar Laplacian inverse in the Schur complement approximation, Hypre or ML should work well, but note that the PCD approximation degrades approximately with the square root of the Reynolds number (so above perhaps Re=100 it starts to struggle).

I also note that the PCD implementation in Firedrake doesn't correctly handle all types of boundary conditions (e.g. it is sub-optimal for inflow or outflow conditions). The correct treatment is described in Section 3 of Jan Blechta's thesis ( https://dspace.cuni.cz/bitstream/handle/20.500.11956/108384/140075745.pdf) in particular the summary in 3.5 is good. We can provide advice on how to do this in the existing PCD framework.

Lawrence

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Dear Zizhou, Along with several collaborators, I have worked on preconditioners for the Navier--Stokes equations in Firedrake. You can see our most recent results in e.g. the tables in https://arxiv.org/abs/2004.09398 . We solved a problem with over one billion degrees of freedom with a low-order discretisation in https://arxiv.org/abs/1810.03315 . You can find our implementation at https://github.com/florianwechsung/alfi . In particular we have a lid-driven cavity set up in 2D and 3D, along with Reynolds-robust solvers for them. You can see some of the command lines that work in examples/Makefile; to start with, look at the command lines with --problem ldc2d. (Solving the Navier-Stokes is hard, and the preconditioner has to change a fair bit with the discretisation employed: the relaxation, prolongation, stabilisation, at least. In alfi you drive all this from the command line.) Hope this helps! Patrick On 12/04/2021 18:42, Ed Bueler wrote:

Zizhou --

The Stokes (not NS) 2D lid-driven cavity example in Chapter 14 of my book focuses on algorithmic and parallel scaling relevant to your NS case.  I look at block and multigrid preconditioner combinations that give good performance at high resolutions.  (The runs shown in the text, done on my desktop, go up to 4 x 10^7 degrees of freedom.)  See https://github.com/bueler/p4pdes/tree/master/python/ch14 for the code, which includes some good solver-option combinations, and here for the book https://my.siam.org/Store/Product/viewproduct/?ProductId=32850137 This may help a little, but as usual no magic bullets.

Ed

On Mon, Apr 12, 2021 at 7:01 AM Zizhou Huang <zizhou@nyu.edu <mailto:zizhou@nyu.edu>> wrote:

To one whom it may concern,

I'm a PhD student at NYU. I'm working on a project on solving the Navier-Stokes equation with FEM, so I'm really interested in the amazing demo of solving the Navier-Stokes equation on the website. (https://www.firedrakeproject.org/demos/navier_stokes.py.html)

I modified the code to solve the 3d lid-driven cavity problem with a uniform grid of size 40x40x40, and found that it takes extremely long time and large memory. I had the same issue with my code of similar sizes, which uses Pardiso to solve the linear system. I'm wondering if fem is indeed unsolvable for a large problem like this? Do you have any experience to make it faster by changing the parameters in the linear solver?

Looking forward to hearing from you.

Best, Zizhou _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake

-- Ed Bueler Dept of Mathematics and Statistics University of Alaska Fairbanks Fairbanks, AK 99775-6660 306C Chapman

_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake