Hi Lawrence, Coming back to this. I have managed to install Firedrake on LANL’s Mustang Machine (Intel Xeon E5-2670) through the firedrake-install script (w/ —developer —no_package_manager —disable_ssh), and it works great on a single compute node (16 cores). However, I am still running into the same issue: the program freezes when I use two or more compute nodes. I even did a simple mpirun -n 2 python myprogram.py where I allocated 2 nodes and 1 ppn. As with my University’s Cluster, I am using their Openmpi-1.6.5 but installed my own Python version, because the system installed python/2.7-anaconda doesn’t have the latest packages (i.e., my install script freezes when I attempt to use the provided python/pip). I copied and pasted 'parameters["pyop2_options"]["check_src_hashes"] = True’ into my code, but it didn’t do anything. Any suggestions or ideas on what I should do? Justin
On Aug 25, 2015, at 3:07 AM, Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 18/08/15 08:38, Justin Chang wrote:
Okay so on one compute node (20 cores, 2 sockets) works fine, even with the warning (originally my code hangs at 20 cores). However, if my sbatch script calls for more than one compute node my program freezes for anything > 20 processes. However, this also happens when I use MPICH. Now I am not sure if it's simply an issue with our university's HPC system or if MPICH also has the same problems as OpenMPI.
Hmm, plausibly something goes wrong somewhere in code generation (such that different processes go down different code paths). Can you try running with:
parameters["pyop2_options"]["check_src_hashes"] = True
Cheers,
Lawrence -----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iQEcBAEBAgAGBQJV3DBbAAoJECOc1kQ8PEYv4Y4IANlMFr5WBIXWIAxTKaTQyACe ymVqIZzF97AxYRpvO5WXTDGTwcPYCWEOlBGoAq9vf6xNqGHBYsoKBf0tSv0gY7bb I7wyWqJvwFRpUsSgvZN5XR7nU0+jMPpaZ5m6m8Md8PLHQnmvErUJjgmnss+Nc//p A4ioYfmK0YNzk+simOb3wAeYFM5lCtvw+8W/1Ll6XeUEyaStzQ9ZDRRLi7DR/Ikq gdz5YvzLrpQHPMVQXnT8NcGxvsIJPXmwMENMsZ9M2lCCE0qcziaIvK7HHjGGTnEP NjvTRWy5PZOpn2r5tPd7T7VYSRIZX3hkW9vTtnhny1Msg1f6EgtszWmHJCxgouY= =NMSC -----END PGP SIGNATURE-----
_______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake