Firedrake Cache files
Hi We noticed that some firedrake users on the Imperial clusters have a large ( between 0.5 M and 1.3M) numbers of files in their cache (usually in firedrake/.cache/pyop2) . Is this cache size configurable? If so can you let people know how to limit it. It has two bad effects; firstly no filesystem is happy with that many files in a single directory and it will slow things down to the point it is useless as cache. Also each user is limited to 10M files and a very large cache will effectively reduce their quota to well below the 1TB default (they will run out of files before they run out of blocks). Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs> [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png]
Hi Bob,
On 14 Mar 2019, at 10:08, Cregan, Bob <b.cregan@imperial.ac.uk> wrote:
Hi We noticed that some firedrake users on the Imperial clusters have a large ( between 0.5 M and 1.3M) numbers of files in their cache (usually in firedrake/.cache/pyop2) .
Oh wow, that's a lot. That number is somewhat surprising to me for two reasons: 1. When users update this cache should be cleared 2. The point of the cache is that the compiled code in it should be reused across runs. Normal models runs might produce a few 100 or low 1000s of files: Running the test suite produces around 22000 (but that covers probably much more than would be expected in any single model). Question: is it the case that although the number of files is very large in firedrake/.cache/pyop2, the number of files in firedrake/.cache/tsfc is much lower? (I'm trying to crystal-ball the likely cause). To clear things out, can you suggest to them that they run "firedrake-clean" (this purges the caches).
Is this cache size configurable? If so can you let people know how to limit it.
Right now it is not. We can add code to do so, but I have two concerns/questions): 1. The number of files you report suggests that somehow either we (or the model users) have a bug somewhere. 2. Where should we cap (and how do we evict). This is more a design decision question for us.
It has two bad effects; firstly no filesystem is happy with that many files in a single directory and it will slow things down to the point it is useless as cache. Also each user is limited to 10M files and a very large cache will effectively reduce their quota to well below the 1TB default (they will run out of files before they run out of blocks).
Sure. If you know who it is, can you ask them to liaise with us to try and figure out what is going on? Cheers, Lawrence
Hi Lawrence There are only a few users I'll get them to contact you. This email, or firedrake@imperial.ac.uk? The number of files in tsfc is smaller but not hugely so 100K compared to 2.2M in one. I won't suggest firedrake-clean to them as I guess you want to do some forensics. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs> [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Lawrence Mitchell <wence@gmx.li> Sent: 14 March 2019 10:40 To: Cregan, Bob Cc: firedrake Subject: Re: [firedrake] Firedrake Cache files Hi Bob,
On 14 Mar 2019, at 10:08, Cregan, Bob <b.cregan@imperial.ac.uk> wrote:
Hi We noticed that some firedrake users on the Imperial clusters have a large ( between 0.5 M and 1.3M) numbers of files in their cache (usually in firedrake/.cache/pyop2) .
Oh wow, that's a lot. That number is somewhat surprising to me for two reasons: 1. When users update this cache should be cleared 2. The point of the cache is that the compiled code in it should be reused across runs. Normal models runs might produce a few 100 or low 1000s of files: Running the test suite produces around 22000 (but that covers probably much more than would be expected in any single model). Question: is it the case that although the number of files is very large in firedrake/.cache/pyop2, the number of files in firedrake/.cache/tsfc is much lower? (I'm trying to crystal-ball the likely cause). To clear things out, can you suggest to them that they run "firedrake-clean" (this purges the caches).
Is this cache size configurable? If so can you let people know how to limit it.
Right now it is not. We can add code to do so, but I have two concerns/questions): 1. The number of files you report suggests that somehow either we (or the model users) have a bug somewhere. 2. Where should we cap (and how do we evict). This is more a design decision question for us.
It has two bad effects; firstly no filesystem is happy with that many files in a single directory and it will slow things down to the point it is useless as cache. Also each user is limited to 10M files and a very large cache will effectively reduce their quota to well below the 1TB default (they will run out of files before they run out of blocks).
Sure. If you know who it is, can you ask them to liaise with us to try and figure out what is going on? Cheers, Lawrence
Hi Bob,
On 14 Mar 2019, at 11:07, Cregan, Bob <b.cregan@imperial.ac.uk> wrote:
Hi Lawrence There are only a few users I'll get them to contact you. This email, or firedrake@imperial.ac.uk?
firedrake@imperial.ac.uk please. That way more eyes see the problem.
The number of files in tsfc is smaller but not hugely so 100K compared to 2.2M in one.
OK, so that provides some likely culprits, but a little more forensics will be required.
I won't suggest firedrake-clean to them as I guess you want to do some forensics.
Good point. Thanks, Lawrence
My crystal ball says that people don't know how to use Constant, so they instead hard-code things that change at each iteration, and that will generate a lot of compiled code. (But it's just a guess...) "Cregan, Bob" <b.cregan@imperial.ac.uk> writes:
Hi
We noticed that some firedrake users on the Imperial clusters have a large ( between 0.5 M and 1.3M) numbers of files in their cache (usually in firedrake/.cache/pyop2) .
Is this cache size configurable? If so can you let people know how to limit it. It has two bad effects; firstly no filesystem is happy with that many files in a single directory and it will slow things down to the point it is useless as cache. Also each user is limited to 10M files and a very large cache will effectively reduce their quota to well below the 1TB default (they will run out of files before they run out of blocks).
Thanks
Bob
Bob Cregan HPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>
[1505984389175_twitter.png] @imperialRCS @imperialRSE
[1505983882959_Imperial-RCS.png]
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Hi, I'm a new user running on my own machine, not the Imperial clusters. Just noticed that I have almost half a Gig of compiled code and whatnot in my .cache generated in just a few hours of tutorials. I would be really interested in how to minimize this output. Could you please email the recommendations to this list? Thanks Ziad On 2019-03-14 11:21 a.m., Miklós Homolya wrote:
My crystal ball says that people don't know how to use Constant, so they instead hard-code things that change at each iteration, and that will generate a lot of compiled code. (But it's just a guess...)
"Cregan, Bob" <b.cregan@imperial.ac.uk> writes:
Hi
We noticed that some firedrake users on the Imperial clusters have a large ( between 0.5 M and 1.3M) numbers of files in their cache (usually in firedrake/.cache/pyop2) .
Is this cache size configurable? If so can you let people know how to limit it. It has two bad effects; firstly no filesystem is happy with that many files in a single directory and it will slow things down to the point it is useless as cache. Also each user is limited to 10M files and a very large cache will effectively reduce their quota to well below the 1TB default (they will run out of files before they run out of blocks).
Thanks
Bob
Bob Cregan HPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>
[1505984389175_twitter.png] @imperialRCS @imperialRSE
[1505983882959_Imperial-RCS.png]
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Hi, Sorry for the delay. I've emailed two people who are affected so they should get in touch with you fairly soon. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs> [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Mikl?s Homolya <miklos.homolya@tum.de> Sent: 14 March 2019 15:21 To: Cregan, Bob; firedrake Subject: Re: [firedrake] Firedrake Cache files My crystal ball says that people don't know how to use Constant, so they instead hard-code things that change at each iteration, and that will generate a lot of compiled code. (But it's just a guess...) "Cregan, Bob" <b.cregan@imperial.ac.uk> writes:
Hi
We noticed that some firedrake users on the Imperial clusters have a large ( between 0.5 M and 1.3M) numbers of files in their cache (usually in firedrake/.cache/pyop2) .
Is this cache size configurable? If so can you let people know how to limit it. It has two bad effects; firstly no filesystem is happy with that many files in a single directory and it will slow things down to the point it is useless as cache. Also each user is limited to 10M files and a very large cache will effectively reduce their quota to well below the 1TB default (they will run out of files before they run out of blocks).
Thanks
Bob
Bob Cregan HPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>>
[1505984389175_twitter.png] @imperialRCS @imperialRSE
[1505983882959_Imperial-RCS.png]
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Hello, I am one such person. However, the guilty directory (with ~350k files) was in an old version of Firedrake, last used a year ago (and last updated even longer ago). For reasons I can't recall, I'd made a fresh Firedrake directory but kept the old one. The new one has ~2k files in firedrake/.cache/pyop2. I'm a Thetis user, so I assume this is a reasonable number. Simon ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Cregan, Bob <b.cregan@imperial.ac.uk> Sent: 19 March 2019 13:06 To: firedrake Subject: Re: [firedrake] Firedrake Cache files Hi, Sorry for the delay. I've emailed two people who are affected so they should get in touch with you fairly soon. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs> [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Miklós Homolya <miklos.homolya@tum.de> Sent: 14 March 2019 15:21 To: Cregan, Bob; firedrake Subject: Re: [firedrake] Firedrake Cache files My crystal ball says that people don't know how to use Constant, so they instead hard-code things that change at each iteration, and that will generate a lot of compiled code. (But it's just a guess...) "Cregan, Bob" <b.cregan@imperial.ac.uk> writes:
Hi
We noticed that some firedrake users on the Imperial clusters have a large ( between 0.5 M and 1.3M) numbers of files in their cache (usually in firedrake/.cache/pyop2) .
Is this cache size configurable? If so can you let people know how to limit it. It has two bad effects; firstly no filesystem is happy with that many files in a single directory and it will slow things down to the point it is useless as cache. Also each user is limited to 10M files and a very large cache will effectively reduce their quota to well below the 1TB default (they will run out of files before they run out of blocks).
Thanks
Bob
Bob Cregan HPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>>
[1505984389175_twitter.png] @imperialRCS @imperialRSE
[1505983882959_Imperial-RCS.png]
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
Hi Simon 2k is fine. It is when you get to the many 10s of K that things start to go awry. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs> [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Warder, Simon C Sent: 20 March 2019 11:22 To: Cregan, Bob; firedrake Subject: Re: [firedrake] Firedrake Cache files Hello, I am one such person. However, the guilty directory (with ~350k files) was in an old version of Firedrake, last used a year ago (and last updated even longer ago). For reasons I can't recall, I'd made a fresh Firedrake directory but kept the old one. The new one has ~2k files in firedrake/.cache/pyop2. I'm a Thetis user, so I assume this is a reasonable number. Simon ________________________________ From: firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> on behalf of Cregan, Bob <b.cregan@imperial.ac.uk> Sent: 19 March 2019 13:06 To: firedrake Subject: Re: [firedrake] Firedrake Cache files Hi, Sorry for the delay. I've emailed two people who are affected so they should get in touch with you fairly soon. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs> [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Miklós Homolya <miklos.homolya@tum.de> Sent: 14 March 2019 15:21 To: Cregan, Bob; firedrake Subject: Re: [firedrake] Firedrake Cache files My crystal ball says that people don't know how to use Constant, so they instead hard-code things that change at each iteration, and that will generate a lot of compiled code. (But it's just a guess...) "Cregan, Bob" <b.cregan@imperial.ac.uk> writes:
Hi
We noticed that some firedrake users on the Imperial clusters have a large ( between 0.5 M and 1.3M) numbers of files in their cache (usually in firedrake/.cache/pyop2) .
Is this cache size configurable? If so can you let people know how to limit it. It has two bad effects; firstly no filesystem is happy with that many files in a single directory and it will slow things down to the point it is useless as cache. Also each user is limited to 10M files and a very large cache will effectively reduce their quota to well below the 1TB default (they will run out of files before they run out of blocks).
Thanks
Bob
Bob Cregan HPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>>
[1505984389175_twitter.png] @imperialRCS @imperialRSE
[1505983882959_Imperial-RCS.png]
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake
participants (5)
-
Cregan, Bob
-
Lawrence Mitchell
-
Miklós Homolya
-
Warder, Simon C
-
Ziad Boutanios