loopy wants to make a directory
Hi folks, When running on ARCHER (national supercomputer) we need to set new paths for writing cache files because you have to build in /home but compute nodes are only allowed to write to /work. Since the loo.py changes it looks like we need to set paths for loo.py as well, otherwise I get the message at the bottom of this email. How do I set them? all the best --cjc import loopy File "/home2/n02/n02/ccotter/firedrake/firedrake/src/loopy/loopy/__init__.py", line 106, in <module> _PersistentDictBase.__init__(self, identifier, key_builder, container_dir) File "/home2/n02/n02/ccotter/firedrake/firedrake/lib/python3.7/site-packages/pytools/persistent_dict.py", line 441, in __init__ key_builder=LoopyKeyBuilder()) File "/home2/n02/n02/ccotter/firedrake/firedrake/lib/python3.7/site-packages/pytools/persistent_dict.py", line 544, in __init__ self._make_container_dir() File "/home2/n02/n02/ccotter/firedrake/firedrake/lib/python3.7/site-packages/pytools/persistent_dict.py", line 486, in _make_container_dir self._make_container_dir() _PersistentDictBase.__init__(self, identifier, key_builder, container_dir) File "/home2/n02/n02/ccotter/firedrake/firedrake/lib/python3.7/site-packages/pytools/persistent_dict.py", line 441, in __init__ File "/home2/n02/n02/ccotter/firedrake/firedrake/lib/python3.7/site-packages/pytools/persistent_dict.py", line 486, in _make_container_dir _make_dir_recursively(self.container_dir) File "/home2/n02/n02/ccotter/firedrake/firedrake/lib/python3.7/site-packages/pytools/persistent_dict.py", line 77, in _make_dir_recursively
Hi Colin, cc'ing the loopy mailing list.
On 6 Jun 2019, at 07:02, Cotter, Colin J <colin.cotter@imperial.ac.uk> wrote:
Hi folks, When running on ARCHER (national supercomputer) we need to set new paths for writing cache files because you have to build in /home but compute nodes are only allowed to write to /work. Since the loo.py changes it looks like we need to set paths for loo.py as well, otherwise I get the message at the bottom of this email. How do I set them?
It looks like loopy (via pytools) sets up these caches by asking python where the use cache directory is. This is controlled by the XDG_CACHE_HOME environment variable Running your firedrake with: export XDG_CACHE_HOME=$VIRTUAL_ENV/.cache Should do the trick (once you've activated the virtualenv before setting the var). Or try this patch? diff --git a/firedrake_configuration/__init__.py b/firedrake_configuration/__init__.py index b64b1b52..c6a1566b 100644 --- a/firedrake_configuration/__init__.py +++ b/firedrake_configuration/__init__.py @@ -46,3 +46,5 @@ def setup_cache_dirs(): os.environ["PYOP2_CACHE_DIR"] = os.path.join(config["options"]["cache_dir"], "pyop2") if 'FIREDRAKE_TSFC_KERNEL_CACHE_DIR' not in os.environ: os.environ["FIREDRAKE_TSFC_KERNEL_CACHE_DIR"] = os.path.join(config["options"]["cache_dir"], "tsfc") + + os.environ["XDG_CACHE_HOME"] = config["options"]["cache_dir"] @loopy folk, I guess you're being good citizens by asking python for the cache directory. But controlling this on supercomputers is potentially an issue: you need to set different things depending on whether you're on linux or not. I guess we can just assume that on a supercomputer you're on linux. Thoughts? Thanks, Lawrence
Lawrence Mitchell <wence@gmx.li> writes:
Hi Colin,
cc'ing the loopy mailing list.
On 6 Jun 2019, at 07:02, Cotter, Colin J <colin.cotter@imperial.ac.uk> wrote:
Hi folks, When running on ARCHER (national supercomputer) we need to set new paths for writing cache files because you have to build in /home but compute nodes are only allowed to write to /work. Since the loo.py changes it looks like we need to set paths for loo.py as well, otherwise I get the message at the bottom of this email. How do I set them?
It looks like loopy (via pytools) sets up these caches by asking python where the use cache directory is. This is controlled by the XDG_CACHE_HOME environment variable
Running your firedrake with:
export XDG_CACHE_HOME=$VIRTUAL_ENV/.cache
Should do the trick (once you've activated the virtualenv before setting the var).
Or try this patch?
diff --git a/firedrake_configuration/__init__.py b/firedrake_configuration/__init__.py index b64b1b52..c6a1566b 100644 --- a/firedrake_configuration/__init__.py +++ b/firedrake_configuration/__init__.py @@ -46,3 +46,5 @@ def setup_cache_dirs(): os.environ["PYOP2_CACHE_DIR"] = os.path.join(config["options"]["cache_dir"], "pyop2") if 'FIREDRAKE_TSFC_KERNEL_CACHE_DIR' not in os.environ: os.environ["FIREDRAKE_TSFC_KERNEL_CACHE_DIR"] = os.path.join(config["options"]["cache_dir"], "tsfc") + + os.environ["XDG_CACHE_HOME"] = config["options"]["cache_dir"]
@loopy folk, I guess you're being good citizens by asking python for the cache directory. But controlling this on supercomputers is potentially an issue: you need to set different things depending on whether you're on linux or not.
At the moment, Loopy follows the "XDG Base Directory Specification" [1]. This (and the appdirs package) is in principle Linux-specific, but both (the spec and the package) technically work on other OSs, which is what is happening. That was the only sane convention that I was able to find at the time the cache code got written. Arguably, even though this is a desktop-focused spec, Linux supercomputer centers would be well-advised to set $XDG_CACHE_HOME to something sane, which might make this problem go away. If you have another convention you'd like pytools to follow, I'm happy to consider it. Another thing that could easily be changed: Right now, a lot of this configuration gets enacted at module import, which potentially makes it hard to change programmatically. I'd be happy to move this to time of first use, so that Firedrake has a chance to set things up beforehand. [1] https://specifications.freedesktop.org/basedir-spec/latest/ HTH, Andreas
Hi Lawrence, Setting XDG_CACHE_HOME works, thanks! We'll update the ARCHER Wiki page. cheers --cjc ________________________________ From: Lawrence Mitchell <wence@gmx.li> Sent: 06 June 2019 09:37:37 To: Cotter, Colin J Cc: firedrake; loopy@tiker.net Subject: Re: [firedrake] loopy wants to make a directory Hi Colin, cc'ing the loopy mailing list.
On 6 Jun 2019, at 07:02, Cotter, Colin J <colin.cotter@imperial.ac.uk> wrote:
Hi folks, When running on ARCHER (national supercomputer) we need to set new paths for writing cache files because you have to build in /home but compute nodes are only allowed to write to /work. Since the loo.py changes it looks like we need to set paths for loo.py as well, otherwise I get the message at the bottom of this email. How do I set them?
It looks like loopy (via pytools) sets up these caches by asking python where the use cache directory is. This is controlled by the XDG_CACHE_HOME environment variable Running your firedrake with: export XDG_CACHE_HOME=$VIRTUAL_ENV/.cache Should do the trick (once you've activated the virtualenv before setting the var). Or try this patch? diff --git a/firedrake_configuration/__init__.py b/firedrake_configuration/__init__.py index b64b1b52..c6a1566b 100644 --- a/firedrake_configuration/__init__.py +++ b/firedrake_configuration/__init__.py @@ -46,3 +46,5 @@ def setup_cache_dirs(): os.environ["PYOP2_CACHE_DIR"] = os.path.join(config["options"]["cache_dir"], "pyop2") if 'FIREDRAKE_TSFC_KERNEL_CACHE_DIR' not in os.environ: os.environ["FIREDRAKE_TSFC_KERNEL_CACHE_DIR"] = os.path.join(config["options"]["cache_dir"], "tsfc") + + os.environ["XDG_CACHE_HOME"] = config["options"]["cache_dir"] @loopy folk, I guess you're being good citizens by asking python for the cache directory. But controlling this on supercomputers is potentially an issue: you need to set different things depending on whether you're on linux or not. I guess we can just assume that on a supercomputer you're on linux. Thoughts? Thanks, Lawrence
participants (3)
-
Andreas Kloeckner
-
Cotter, Colin J
-
Lawrence Mitchell