Hi Lawrence There are only a few users I'll get them to contact you. This email, or firedrake@imperial.ac.uk? The number of files in tsfc is smaller but not hugely so 100K compared to 2.2M in one. I won't suggest firedrake-clean to them as I guess you want to do some forensics. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan@imperial.ac.uk W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs> [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Lawrence Mitchell <wence@gmx.li> Sent: 14 March 2019 10:40 To: Cregan, Bob Cc: firedrake Subject: Re: [firedrake] Firedrake Cache files Hi Bob,
On 14 Mar 2019, at 10:08, Cregan, Bob <b.cregan@imperial.ac.uk> wrote:
Hi We noticed that some firedrake users on the Imperial clusters have a large ( between 0.5 M and 1.3M) numbers of files in their cache (usually in firedrake/.cache/pyop2) .
Oh wow, that's a lot. That number is somewhat surprising to me for two reasons: 1. When users update this cache should be cleared 2. The point of the cache is that the compiled code in it should be reused across runs. Normal models runs might produce a few 100 or low 1000s of files: Running the test suite produces around 22000 (but that covers probably much more than would be expected in any single model). Question: is it the case that although the number of files is very large in firedrake/.cache/pyop2, the number of files in firedrake/.cache/tsfc is much lower? (I'm trying to crystal-ball the likely cause). To clear things out, can you suggest to them that they run "firedrake-clean" (this purges the caches).
Is this cache size configurable? If so can you let people know how to limit it.
Right now it is not. We can add code to do so, but I have two concerns/questions): 1. The number of files you report suggests that somehow either we (or the model users) have a bug somewhere. 2. Where should we cap (and how do we evict). This is more a design decision question for us.
It has two bad effects; firstly no filesystem is happy with that many files in a single directory and it will slow things down to the point it is useless as cache. Also each user is limited to 10M files and a very large cache will effectively reduce their quota to well below the 1TB default (they will run out of files before they run out of blocks).
Sure. If you know who it is, can you ask them to liaise with us to try and figure out what is going on? Cheers, Lawrence
