Re: [firedrake] Firedrake Cache files
The usual cause is having a UFL form which contains a rapidly-changing floating point number. This is fine: solve(inner(grad(u), grad(v))*dx = 2.0*f*v*dx, out, ...) # solve -div(grad(u)) = 2*f just once This is a bad idea: t = 0.0; T = 500.0; dt = 0.1 while t < T: solve(inner(grad(u), grad(v))*dx = sin(t)*f*v*dx, out, ...) # multiply forcing by sin(t) t += dt This would generate and compile 5000 different C files, with the values t = 0.0, 0.1, 0.2, 0.3, ... each "hardcoded" into the C code of one form. A better solution is to use a Constant object inside the form, and update this within the loop: t = 0.0; T = 500.0; dt = 0.1 tc = Constant(t) while t < T: solve(inner(grad(u), grad(v))*dx = sin(tc)*f*v*dx, out, ...) # multiply forcing by sin(t) t += dt tc.assign(t) This will generate a single compiled form where the current time is passed in via a pointer, rather than being hardcoded into the C code. Note: if you are repeatedly solving a problem like this, it's a bit better to use the LinearVariationalProblem/LinearVariationalSolver objects, as described at https://www.firedrakeproject.org/demos/DG_advection.py.html, but this is ultimately separate to Constant vs hard-coded floating point numbers. On Thu, 14 Mar 2019 at 18:57, Ziad Boutanios <ziad.boutanios@gmail.com> wrote:
Hi,
I'm a new user running on my own machine, not the Imperial clusters. Just noticed that I have almost half a Gig of compiled code and whatnot in my .cache generated in just a few hours of tutorials. I would be really interested in how to minimize this output. Could you please email the recommendations to this list?
Thanks
Ziad
On 2019-03-14 11:21 a.m., Miklós Homolya wrote:
My crystal ball says that people don't know how to use Constant, so they instead hard-code things that change at each iteration, and that will generate a lot of compiled code. (But it's just a guess...)
"Cregan, Bob" <b.cregan@imperial.ac.uk> writes:
Hi
We noticed that some firedrake users on the Imperial clusters have a large ( between 0.5 M and 1.3M) numbers of files in their cache (usually in firedrake/.cache/pyop2) .
Is this cache size configurable? If so can you let people know how to limit it. It has two bad effects; firstly no filesystem is happy with that many files in a single directory and it will slow things down to the point it is useless as cache. Also each user is limited to 10M files and a very large cache will effectively reduce their quota to well below the 1TB default (they will run out of files before they run out of blocks).
Thanks
Bob
Bob Cregan HPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>
[1505984389175_twitter.png] @imperialRCS @imperialRSE
[1505983882959_Imperial-RCS.png]
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
I understand this is how it currently works, but this is an easy "mistake" to make especially in a complex model with lots of forcing terms. Would it not be possible to always treat a float as a parameter in the form, i.e. don't include the value of the float in the hash for the compiled code, and substitute the right value at run-time? The form itself based on a specific float, could still be treated as immutable everywhere else in firedrake. Admittedly, I have no idea how easy it is to do this Cheers Stephan On 14/03/2019 20:10, Andrew McRae wrote:
The usual cause is having a UFL form which contains a rapidly-changing floating point number.
This is fine: solve(inner(grad(u), grad(v))*dx = 2.0*f*v*dx, out, ...) # solve -div(grad(u)) = 2*f just once
This is a bad idea: t = 0.0; T = 500.0; dt = 0.1 while t < T: solve(inner(grad(u), grad(v))*dx = sin(t)*f*v*dx, out, ...) # multiply forcing by sin(t) t += dt
This would generate and compile 5000 different C files, with the values t = 0.0, 0.1, 0.2, 0.3, ... each "hardcoded" into the C code of one form.
A better solution is to use a Constant object inside the form, and update this within the loop: t = 0.0; T = 500.0; dt = 0.1 tc = Constant(t) while t < T: solve(inner(grad(u), grad(v))*dx = sin(tc)*f*v*dx, out, ...) # multiply forcing by sin(t) t += dt tc.assign(t)
This will generate a single compiled form where the current time is passed in via a pointer, rather than being hardcoded into the C code.
Note: if you are repeatedly solving a problem like this, it's a bit better to use the LinearVariationalProblem/LinearVariationalSolver objects, as described at https://www.firedrakeproject.org/demos/DG_advection.py.html, but this is ultimately separate to Constant vs hard-coded floating point numbers.
On Thu, 14 Mar 2019 at 18:57, Ziad Boutanios <ziad.boutanios@gmail.com <mailto:ziad.boutanios@gmail.com>> wrote:
Hi,
I'm a new user running on my own machine, not the Imperial clusters. Just noticed that I have almost half a Gig of compiled code and whatnot in my .cache generated in just a few hours of tutorials. I would be really interested in how to minimize this output. Could you please email the recommendations to this list?
Thanks
Ziad
On 2019-03-14 11:21 a.m., Miklós Homolya wrote: > My crystal ball says that people don't know how to use Constant, so they > instead hard-code things that change at each iteration, and that will > generate a lot of compiled code. (But it's just a guess...) > > "Cregan, Bob" <b.cregan@imperial.ac.uk <mailto:b.cregan@imperial.ac.uk>> writes: >> Hi >> >> We noticed that some firedrake users on the Imperial clusters have >> a large ( between 0.5 M and 1.3M) numbers of files in their cache >> (usually in firedrake/.cache/pyop2) . >> >> Is this cache size configurable? If so can you let people know how to >> limit it. It has two bad effects; firstly no filesystem is happy with >> that many files in a single directory and it will slow things down to >> the point it is useless as cache. Also each user is limited to 10M >> files and a very large cache will effectively reduce their quota to >> well below the 1TB default (they will run out of files before they run >> out of blocks). >> >> Thanks >> >> Bob >> >> >> >> Bob Cregan >> HPC Systems Analyst >> >> Information & Communication Technologies >> >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan@imperial.ac.uk <mailto:b.cregan@imperial.ac.uk> >> >> W: www.imperial.ac.uk/ict/rcs <http://www.imperial.ac.uk/ict/rcs><http://www.imperial.ac.uk/ict/rcs> >> >> [1505984389175_twitter.png] @imperialRCS @imperialRSE >> >> [1505983882959_Imperial-RCS.png] >> >> _______________________________________________ >> firedrake mailing list >> firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake > _______________________________________________ > firedrake mailing list > firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> > https://mailman.ic.ac.uk/mailman/listinfo/firedrake
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Let’s discuss this in the Thetis meeting Sent from my iPhone
On 15 Mar 2019, at 12:00, Stephan Kramer <s.kramer@imperial.ac.uk> wrote:
I understand this is how it currently works, but this is an easy "mistake" to make especially in a complex model with lots of forcing terms. Would it not be possible to always treat a float as a parameter in the form, i.e. don't include the value of the float in the hash for the compiled code, and substitute the right value at run-time? The form itself based on a specific float, could still be treated as immutable everywhere else in firedrake. Admittedly, I have no idea how easy it is to do this
Cheers Stephan
On 14/03/2019 20:10, Andrew McRae wrote: The usual cause is having a UFL form which contains a rapidly-changing floating point number. This is fine: solve(inner(grad(u), grad(v))*dx = 2.0*f*v*dx, out, ...) # solve -div(grad(u)) = 2*f just once This is a bad idea: t = 0.0; T = 500.0; dt = 0.1 while t < T: solve(inner(grad(u), grad(v))*dx = sin(t)*f*v*dx, out, ...) # multiply forcing by sin(t) t += dt This would generate and compile 5000 different C files, with the values t = 0.0, 0.1, 0.2, 0.3, ... each "hardcoded" into the C code of one form. A better solution is to use a Constant object inside the form, and update this within the loop: t = 0.0; T = 500.0; dt = 0.1 tc = Constant(t) while t < T: solve(inner(grad(u), grad(v))*dx = sin(tc)*f*v*dx, out, ...) # multiply forcing by sin(t) t += dt tc.assign(t) This will generate a single compiled form where the current time is passed in via a pointer, rather than being hardcoded into the C code. Note: if you are repeatedly solving a problem like this, it's a bit better to use the LinearVariationalProblem/LinearVariationalSolver objects, as described at https://www.firedrakeproject.org/demos/DG_advection.py.html, but this is ultimately separate to Constant vs hard-coded floating point numbers. On Thu, 14 Mar 2019 at 18:57, Ziad Boutanios <ziad.boutanios@gmail.com <mailto:ziad.boutanios@gmail.com>> wrote: Hi, I'm a new user running on my own machine, not the Imperial clusters. Just noticed that I have almost half a Gig of compiled code and whatnot in my .cache generated in just a few hours of tutorials. I would be really interested in how to minimize this output. Could you please email the recommendations to this list? Thanks Ziad On 2019-03-14 11:21 a.m., Miklós Homolya wrote: > My crystal ball says that people don't know how to use Constant, so they > instead hard-code things that change at each iteration, and that will > generate a lot of compiled code. (But it's just a guess...) > > "Cregan, Bob" <b.cregan@imperial.ac.uk <mailto:b.cregan@imperial.ac.uk>> writes: >> Hi >> >> We noticed that some firedrake users on the Imperial clusters have >> a large ( between 0.5 M and 1.3M) numbers of files in their cache >> (usually in firedrake/.cache/pyop2) . >> >> Is this cache size configurable? If so can you let people know how to >> limit it. It has two bad effects; firstly no filesystem is happy with >> that many files in a single directory and it will slow things down to >> the point it is useless as cache. Also each user is limited to 10M >> files and a very large cache will effectively reduce their quota to >> well below the 1TB default (they will run out of files before they run >> out of blocks). >> >> Thanks >> >> Bob >> >> >> >> Bob Cregan >> HPC Systems Analyst >> >> Information & Communication Technologies >> >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan@imperial.ac.uk <mailto:b.cregan@imperial.ac.uk> >> >> W: www.imperial.ac.uk/ict/rcs <http://www.imperial.ac.uk/ict/rcs><http://www.imperial.ac.uk/ict/rcs> >> >> [1505984389175_twitter.png] @imperialRCS @imperialRSE >> >> [1505983882959_Imperial-RCS.png] >> >> _______________________________________________ >> firedrake mailing list >> firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake > _______________________________________________ > firedrake mailing list > firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> > https://mailman.ic.ac.uk/mailman/listinfo/firedrake _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Arguably one could let some logging through when just-in-time compilation happens, although that typically evokes the wrong reaction from users: "firedrake keeps printing all this crap! how to disable it?" I find it a pretty aggressive move to convert all floats to Constants, since it easily leads to a perfusion of kernel arguments. Though, it may still be the safest default behaviour, but I think one should at least be able to disable it. Stephan Kramer <s.kramer@imperial.ac.uk> writes:
I understand this is how it currently works, but this is an easy "mistake" to make especially in a complex model with lots of forcing terms. Would it not be possible to always treat a float as a parameter in the form, i.e. don't include the value of the float in the hash for the compiled code, and substitute the right value at run-time? The form itself based on a specific float, could still be treated as immutable everywhere else in firedrake. Admittedly, I have no idea how easy it is to do this
Cheers Stephan
On 14/03/2019 20:10, Andrew McRae wrote:
The usual cause is having a UFL form which contains a rapidly-changing floating point number.
This is fine: solve(inner(grad(u), grad(v))*dx = 2.0*f*v*dx, out, ...) # solve -div(grad(u)) = 2*f just once
This is a bad idea: t = 0.0; T = 500.0; dt = 0.1 while t < T: solve(inner(grad(u), grad(v))*dx = sin(t)*f*v*dx, out, ...) # multiply forcing by sin(t) t += dt
This would generate and compile 5000 different C files, with the values t = 0.0, 0.1, 0.2, 0.3, ... each "hardcoded" into the C code of one form.
A better solution is to use a Constant object inside the form, and update this within the loop: t = 0.0; T = 500.0; dt = 0.1 tc = Constant(t) while t < T: solve(inner(grad(u), grad(v))*dx = sin(tc)*f*v*dx, out, ...) # multiply forcing by sin(t) t += dt tc.assign(t)
This will generate a single compiled form where the current time is passed in via a pointer, rather than being hardcoded into the C code.
Note: if you are repeatedly solving a problem like this, it's a bit better to use the LinearVariationalProblem/LinearVariationalSolver objects, as described at https://www.firedrakeproject.org/demos/DG_advection.py.html, but this is ultimately separate to Constant vs hard-coded floating point numbers.
On Thu, 14 Mar 2019 at 18:57, Ziad Boutanios <ziad.boutanios@gmail.com <mailto:ziad.boutanios@gmail.com>> wrote:
Hi,
I'm a new user running on my own machine, not the Imperial clusters. Just noticed that I have almost half a Gig of compiled code and whatnot in my .cache generated in just a few hours of tutorials. I would be really interested in how to minimize this output. Could you please email the recommendations to this list?
Thanks
Ziad
On 2019-03-14 11:21 a.m., Miklós Homolya wrote: > My crystal ball says that people don't know how to use Constant, so they > instead hard-code things that change at each iteration, and that will > generate a lot of compiled code. (But it's just a guess...) > > "Cregan, Bob" <b.cregan@imperial.ac.uk <mailto:b.cregan@imperial.ac.uk>> writes: >> Hi >> >> We noticed that some firedrake users on the Imperial clusters have >> a large ( between 0.5 M and 1.3M) numbers of files in their cache >> (usually in firedrake/.cache/pyop2) . >> >> Is this cache size configurable? If so can you let people know how to >> limit it. It has two bad effects; firstly no filesystem is happy with >> that many files in a single directory and it will slow things down to >> the point it is useless as cache. Also each user is limited to 10M >> files and a very large cache will effectively reduce their quota to >> well below the 1TB default (they will run out of files before they run >> out of blocks). >> >> Thanks >> >> Bob >> >> >> >> Bob Cregan >> HPC Systems Analyst >> >> Information & Communication Technologies >> >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan@imperial.ac.uk <mailto:b.cregan@imperial.ac.uk> >> >> W: www.imperial.ac.uk/ict/rcs <http://www.imperial.ac.uk/ict/rcs><http://www.imperial.ac.uk/ict/rcs> >> >> [1505984389175_twitter.png] @imperialRCS @imperialRSE >> >> [1505983882959_Imperial-RCS.png] >> >> _______________________________________________ >> firedrake mailing list >> firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake > _______________________________________________ > firedrake mailing list > firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> > https://mailman.ic.ac.uk/mailman/listinfo/firedrake
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
An improved version of the logging idea would be this: Stay silent for the first, say, few hundred to a thousand invocations of compiling generated code, but then start emitting visible warnings at each invocation. The message of the warning would warn for the potential issue of generating code in each time stepping iteration, and would reference the corresponding FAQ entry of the documentation, which would describe how to verify if that is the case, how to fix the problem, or to configure the number of swallowed invocations (for very complex models), or even to disable these warnings completely (not recommended). Miklós Homolya <miklos.homolya@tum.de> writes:
Arguably one could let some logging through when just-in-time compilation happens, although that typically evokes the wrong reaction from users: "firedrake keeps printing all this crap! how to disable it?"
I find it a pretty aggressive move to convert all floats to Constants, since it easily leads to a perfusion of kernel arguments. Though, it may still be the safest default behaviour, but I think one should at least be able to disable it.
Stephan Kramer <s.kramer@imperial.ac.uk> writes:
I understand this is how it currently works, but this is an easy "mistake" to make especially in a complex model with lots of forcing terms. Would it not be possible to always treat a float as a parameter in the form, i.e. don't include the value of the float in the hash for the compiled code, and substitute the right value at run-time? The form itself based on a specific float, could still be treated as immutable everywhere else in firedrake. Admittedly, I have no idea how easy it is to do this
Cheers Stephan
On 14/03/2019 20:10, Andrew McRae wrote:
The usual cause is having a UFL form which contains a rapidly-changing floating point number.
This is fine: solve(inner(grad(u), grad(v))*dx = 2.0*f*v*dx, out, ...) # solve -div(grad(u)) = 2*f just once
This is a bad idea: t = 0.0; T = 500.0; dt = 0.1 while t < T: solve(inner(grad(u), grad(v))*dx = sin(t)*f*v*dx, out, ...) # multiply forcing by sin(t) t += dt
This would generate and compile 5000 different C files, with the values t = 0.0, 0.1, 0.2, 0.3, ... each "hardcoded" into the C code of one form.
A better solution is to use a Constant object inside the form, and update this within the loop: t = 0.0; T = 500.0; dt = 0.1 tc = Constant(t) while t < T: solve(inner(grad(u), grad(v))*dx = sin(tc)*f*v*dx, out, ...) # multiply forcing by sin(t) t += dt tc.assign(t)
This will generate a single compiled form where the current time is passed in via a pointer, rather than being hardcoded into the C code.
Note: if you are repeatedly solving a problem like this, it's a bit better to use the LinearVariationalProblem/LinearVariationalSolver objects, as described at https://www.firedrakeproject.org/demos/DG_advection.py.html, but this is ultimately separate to Constant vs hard-coded floating point numbers.
On Thu, 14 Mar 2019 at 18:57, Ziad Boutanios <ziad.boutanios@gmail.com <mailto:ziad.boutanios@gmail.com>> wrote:
Hi,
I'm a new user running on my own machine, not the Imperial clusters. Just noticed that I have almost half a Gig of compiled code and whatnot in my .cache generated in just a few hours of tutorials. I would be really interested in how to minimize this output. Could you please email the recommendations to this list?
Thanks
Ziad
On 2019-03-14 11:21 a.m., Miklós Homolya wrote: > My crystal ball says that people don't know how to use Constant, so they > instead hard-code things that change at each iteration, and that will > generate a lot of compiled code. (But it's just a guess...) > > "Cregan, Bob" <b.cregan@imperial.ac.uk <mailto:b.cregan@imperial.ac.uk>> writes: >> Hi >> >> We noticed that some firedrake users on the Imperial clusters have >> a large ( between 0.5 M and 1.3M) numbers of files in their cache >> (usually in firedrake/.cache/pyop2) . >> >> Is this cache size configurable? If so can you let people know how to >> limit it. It has two bad effects; firstly no filesystem is happy with >> that many files in a single directory and it will slow things down to >> the point it is useless as cache. Also each user is limited to 10M >> files and a very large cache will effectively reduce their quota to >> well below the 1TB default (they will run out of files before they run >> out of blocks). >> >> Thanks >> >> Bob >> >> >> >> Bob Cregan >> HPC Systems Analyst >> >> Information & Communication Technologies >> >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan@imperial.ac.uk <mailto:b.cregan@imperial.ac.uk> >> >> W: www.imperial.ac.uk/ict/rcs <http://www.imperial.ac.uk/ict/rcs><http://www.imperial.ac.uk/ict/rcs> >> >> [1505984389175_twitter.png] @imperialRCS @imperialRSE >> >> [1505983882959_Imperial-RCS.png] >> >> _______________________________________________ >> firedrake mailing list >> firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake > _______________________________________________ > firedrake mailing list > firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> > https://mailman.ic.ac.uk/mailman/listinfo/firedrake
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
On 17 Mar 2019, at 12:20, Miklós Homolya <miklos.homolya@tum.de> wrote:
An improved version of the logging idea would be this:
Stay silent for the first, say, few hundred to a thousand invocations of compiling generated code, but then start emitting visible warnings at each invocation. The message of the warning would warn for the potential issue of generating code in each time stepping iteration, and would reference the corresponding FAQ entry of the documentation, which would describe how to verify if that is the case, how to fix the problem, or to configure the number of swallowed invocations (for very complex models), or even to disable these warnings completely (not recommended).
Miklós Homolya <miklos.homolya@tum.de> writes:
Arguably one could let some logging through when just-in-time compilation happens, although that typically evokes the wrong reaction from users: "firedrake keeps printing all this crap! how to disable it?"
I find it a pretty aggressive move to convert all floats to Constants, since it easily leads to a perfusion of kernel arguments. Though, it may still be the safest default behaviour, but I think one should at least be able to disable it.
Stephan Kramer <s.kramer@imperial.ac.uk> writes:
I understand this is how it currently works, but this is an easy "mistake" to make especially in a complex model with lots of forcing terms. Would it not be possible to always treat a float as a parameter in the form, i.e. don't include the value of the float in the hash for the compiled code, and substitute the right value at run-time? The form itself based on a specific float, could still be treated as immutable everywhere else in firedrake. Admittedly, I have no idea how easy it is to do this
I had a go at making literal-value -> Constant at form creation time. This didn't work well. I suppose one could instead rewrite inside the form_compiler interface (when you actually need the code). I like (additionally) Miklós' idea of logging in some increasingly loud manner. Lawrence
Hi all, I have another variant on this to suggest. How about we just put the check in the firedrake import (on MPI rank 0). We check for having what seems to be an excessive number of cached files (say, over 10k) and spew a warning: "Your Firedrake disk caches contain n files. You should consider clearing them using: firedrake-clean If this issue recurs frequently, it may indicate a problem with the way you are specifying your differential equations in UFL. Please see <url> for more details." What do people think of that? David On 17/03/2019, 12:21, "firedrake-bounces@imperial.ac.uk on behalf of Miklós Homolya" <firedrake-bounces@imperial.ac.uk on behalf of miklos.homolya@tum.de> wrote: An improved version of the logging idea would be this: Stay silent for the first, say, few hundred to a thousand invocations of compiling generated code, but then start emitting visible warnings at each invocation. The message of the warning would warn for the potential issue of generating code in each time stepping iteration, and would reference the corresponding FAQ entry of the documentation, which would describe how to verify if that is the case, how to fix the problem, or to configure the number of swallowed invocations (for very complex models), or even to disable these warnings completely (not recommended). Miklós Homolya <miklos.homolya@tum.de> writes: > Arguably one could let some logging through when just-in-time > compilation happens, although that typically evokes the wrong reaction > from users: "firedrake keeps printing all this crap! how to disable it?" > > I find it a pretty aggressive move to convert all floats to Constants, > since it easily leads to a perfusion of kernel arguments. Though, it > may still be the safest default behaviour, but I think one should at > least be able to disable it. > > Stephan Kramer <s.kramer@imperial.ac.uk> writes: >> I understand this is how it currently works, but this is an easy >> "mistake" to make especially in a complex model with lots of forcing >> terms. Would it not be possible to always treat a float as a parameter >> in the form, i.e. don't include the value of the float in the hash for >> the compiled code, and substitute the right value at run-time? The form >> itself based on a specific float, could still be treated as immutable >> everywhere else in firedrake. Admittedly, I have no idea how easy it is >> to do this >> >> Cheers >> Stephan >> >> >> On 14/03/2019 20:10, Andrew McRae wrote: >>> The usual cause is having a UFL form which contains a rapidly-changing >>> floating point number. >>> >>> This is fine: >>> solve(inner(grad(u), grad(v))*dx = 2.0*f*v*dx, out, ...) # solve >>> -div(grad(u)) = 2*f just once >>> >>> This is a bad idea: >>> t = 0.0; T = 500.0; dt = 0.1 >>> while t < T: >>> solve(inner(grad(u), grad(v))*dx = sin(t)*f*v*dx, out, ...) # >>> multiply forcing by sin(t) >>> t += dt >>> >>> This would generate and compile 5000 different C files, with the values >>> t = 0.0, 0.1, 0.2, 0.3, ... each "hardcoded" into the C code of one form. >>> >>> A better solution is to use a Constant object inside the form, and >>> update this within the loop: >>> t = 0.0; T = 500.0; dt = 0.1 >>> tc = Constant(t) >>> while t < T: >>> solve(inner(grad(u), grad(v))*dx = sin(tc)*f*v*dx, out, ...) # >>> multiply forcing by sin(t) >>> t += dt >>> tc.assign(t) >>> >>> This will generate a single compiled form where the current time is >>> passed in via a pointer, rather than being hardcoded into the C code. >>> >>> Note: if you are repeatedly solving a problem like this, it's a bit >>> better to use the LinearVariationalProblem/LinearVariationalSolver >>> objects, as described at >>> https://www.firedrakeproject.org/demos/DG_advection.py.html, but this is >>> ultimately separate to Constant vs hard-coded floating point numbers. >>> >>> On Thu, 14 Mar 2019 at 18:57, Ziad Boutanios <ziad.boutanios@gmail.com >>> <mailto:ziad.boutanios@gmail.com>> wrote: >>> >>> Hi, >>> >>> I'm a new user running on my own machine, not the Imperial clusters. >>> Just noticed that I have almost half a Gig of compiled code and whatnot >>> in my .cache generated in just a few hours of tutorials. I would be >>> really interested in how to minimize this output. Could you please >>> email >>> the recommendations to this list? >>> >>> Thanks >>> >>> Ziad >>> >>> On 2019-03-14 11:21 a.m., Miklós Homolya wrote: >>> > My crystal ball says that people don't know how to use Constant, >>> so they >>> > instead hard-code things that change at each iteration, and that will >>> > generate a lot of compiled code. (But it's just a guess...) >>> > >>> > "Cregan, Bob" <b.cregan@imperial.ac.uk >>> <mailto:b.cregan@imperial.ac.uk>> writes: >>> >> Hi >>> >> >>> >> We noticed that some firedrake users on the Imperial >>> clusters have >>> >> a large ( between 0.5 M and 1.3M) numbers of files in their cache >>> >> (usually in firedrake/.cache/pyop2) . >>> >> >>> >> Is this cache size configurable? If so can you let people know >>> how to >>> >> limit it. It has two bad effects; firstly no filesystem is >>> happy with >>> >> that many files in a single directory and it will slow things >>> down to >>> >> the point it is useless as cache. Also each user is limited to 10M >>> >> files and a very large cache will effectively reduce their quota to >>> >> well below the 1TB default (they will run out of files before >>> they run >>> >> out of blocks). >>> >> >>> >> Thanks >>> >> >>> >> Bob >>> >> >>> >> >>> >> >>> >> Bob Cregan >>> >> HPC Systems Analyst >>> >> >>> >> Information & Communication Technologies >>> >> >>> >> Imperial College London, >>> >> South Kensington Campus London, SW7 2AZ >>> >> T: 07712388129 >>> >> E: b.cregan@imperial.ac.uk <mailto:b.cregan@imperial.ac.uk> >>> >> >>> >> W: www.imperial.ac.uk/ict/rcs >>> <http://www.imperial.ac.uk/ict/rcs><http://www.imperial.ac.uk/ict/rcs> >>> >> >>> >> [1505984389175_twitter.png] @imperialRCS @imperialRSE >>> >> >>> >> [1505983882959_Imperial-RCS.png] >>> >> >>> >> _______________________________________________ >>> >> firedrake mailing list >>> >> firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> >>> >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>> > _______________________________________________ >>> > firedrake mailing list >>> > firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> >>> > https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>> >>> _______________________________________________ >>> firedrake mailing list >>> firedrake@imperial.ac.uk <mailto:firedrake@imperial.ac.uk> >>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>> >>> >>> _______________________________________________ >>> firedrake mailing list >>> firedrake@imperial.ac.uk >>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake >>> >> >> >> >> _______________________________________________ >> firedrake mailing list >> firedrake@imperial.ac.uk >> https://mailman.ic.ac.uk/mailman/listinfo/firedrake _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
I think this is also a workable idea. "Ham, David A" <david.ham@imperial.ac.uk> writes:
Hi all,
I have another variant on this to suggest. How about we just put the check in the firedrake import (on MPI rank 0). We check for having what seems to be an excessive number of cached files (say, over 10k) and spew a warning:
"Your Firedrake disk caches contain n files. You should consider clearing them using:
firedrake-clean
If this issue recurs frequently, it may indicate a problem with the way you are specifying your differential equations in UFL. Please see <url> for more details."
What do people think of that?
David
On 17/03/2019, 12:21, "firedrake-bounces@imperial.ac.uk on behalf of Miklós Homolya" <firedrake-bounces@imperial.ac.uk on behalf of miklos.homolya@tum.de> wrote:
An improved version of the logging idea would be this:
Stay silent for the first, say, few hundred to a thousand invocations of compiling generated code, but then start emitting visible warnings at each invocation. The message of the warning would warn for the potential issue of generating code in each time stepping iteration, and would reference the corresponding FAQ entry of the documentation, which would describe how to verify if that is the case, how to fix the problem, or to configure the number of swallowed invocations (for very complex models), or even to disable these warnings completely (not recommended).
Miklós Homolya <miklos.homolya@tum.de> writes: > Arguably one could let some logging through when just-in-time > compilation happens, although that typically evokes the wrong reaction > from users: "firedrake keeps printing all this crap! how to disable it?" > > I find it a pretty aggressive move to convert all floats to Constants, > since it easily leads to a perfusion of kernel arguments. Though, it > may still be the safest default behaviour, but I think one should at > least be able to disable it.
participants (5)
-
Andrew McRae
-
Ham, David A
-
Lawrence Mitchell
-
Miklós Homolya
-
Stephan Kramer