Hi Florian,

thanks for your email and suggestions. Here is some more data:

    

      
These mesh sizes should all be manageable afaict, I have run on larger
meshes (up to ~60M cells afair).



      

        
In particular, I was not able to run on 48 cores (i.e. 2 full nodes)

1 node, 16 cores/node = 16 cores, refcount_coarse=5:  Runs
2 nores, 16 cores/node = 48 cores, refcount_coarse=4: hangs without
printing error message


      

      
I presume you mean 24 cores/node? How long did this run for? Was it
eventually killed by the scheduler?


    

    Yes, sorry, should be 24 cores/node for refcount_coarse=4,5.
    However, for refcount=6 I ran on 16 cores/node, so 32 cores in
    total.

    

    Can't tell exactly how long the refcount_coarse=4 case ran until
    crashed, but the wallclock time limit was set to 5 minutes, and I'm
    pretty sure it ran at least for a couple of minutes, i.e. did not
    abort immediately.

    

      

        
2 nores, 16 cores/node = 48 cores, refcount_coarse=5: crashes with PETSc
error message


      

      
This isn't a PETSc error, it's only PETSc reporting the crash. The
actual issue is a compile error. Can you check whether
/work/n02/n02/eike//pyop2-cache/9b5f79130a429e4044a23dacb78191df.so or
.so.tmp exists?


    

    Ah, good to know it's not a PETSc error, I was worried that I had to
    rebuild PETSc... But why does PETSc report the crash if it is an
    issue with the compilation? 

    

    

      

      

        
2 nores, 16 cores/node = 48 cores, refcount_coarse=5: crashes with
different error message


      

      
This is actually the same error. Does
/work/n02/n02/eike//pyop2-cache/c8aac6eec48490d587593472c04a5557.so or
.so.tmp exist?



    

    Can't check if the .so, .so.tmp files exist, but I just repeated a
    run with refcount_coarse=5 on 48 cores (2 full ARCHER nodes). As
    before, it crashes. I attach all files generated by this run, as
    well as the submission scripts as a .tgz file to this email.

    

    For this I also copied the files
    478ec26b47996566adb89c5545f64b5c.{c,err,log} for the pyop2-cache
    directory, for this run no .so files were generated as far as I can
    see.

    The .err file is empty.

    

    I manually compiled the code on a login node by copying the contents
    of 478ec26b47996566adb89c5545f64b5c.log. This works as it should and
    produces 478ec26b47996566adb89c5545f64b5c.so.tmp, but only if I have
    sourced my firedrake.env. Is it possible that this does not get
    loaded properly in some cases? But then it should also crash for
    smaller core counts.

    

    

      

        
Output logs for all cases are attached, as well as the pbs scripts for
running on 48 cores.


      

      
The job script appears to be for a 32 core run. It looks legit, though
you wouldn't need to load the firedrake module given that you also load
the dependency modules and use your own PyOP2/Firedrake


    

    

    Just to clarify, you mean that I can remove 

    

    module load firedrake

    

    but should leave

    

    module load fdrake-build-env

    module load fdrake-python-env

    

    in? I just removed the "module load firedrake" and it does not make
    any difference (but I guess you were not implying that it would fix
    the problem anyway).

    

      

      

        
I also tried running the poisson example in firedrake-benchmark in
different configurations:

1 node, 16 cores/node = 16 cores => works
2 nodes, 8 cores/node = 16 cores => works
2 nodes, 24 cores/node = 48 cores => crashes with PETSc error


      

      
This one is baffling: it appears that only ranks 24-47 are aborted. I
have absolutely no clue what is going on. Is this reproducible?

    

    

    I will look at this again.

    

      



      

        
I did use the generic ARCHER firedrake installation in this case,
though. In the Helmholtz solver runs I used my own version of firedrake
and PyOP2 since I need the multigrid and local_par-loop branches.

This makes me suspicious that it is not a problem specific to my code,
but then I thought that you did run firedrake on relatively large core
counts before, didn't you?


      

      
I have run on up to 1536 cores for Poisson and Cahn-Hilliard.



      

        
The output of the Poisson benchmarks is also attached. 

Any ideas what might be going wrong and what is the best way for
debugging this?


      

      
I'm not entirely sure what to suggest about the run that just hangs
other than trying to pinpoint where exactly that happens.

For the compilation failures, I recall I had those sporadically and
could never figure out what exactly had gone wrong. Check what is in the
cache and also what the compiled code represents: and assembly or an
expression?


    

    

    For the Helmholtz solver run above it seems to be an assembly (see
    output.log):

    

    assemble(self.psi*div(self.dMinvMr_u)*dx))

    

    Cheers,

    

    Eike

    

    

    

      
Florian



      

        
Thanks,

Eike


      

      

      

      

      

      
_______________________________________________
firedrake mailing list
firedrake@imperial.ac.uk
https://mailman.ic.ac.uk/mailman/listinfo/firedrake


    

    

    
-- 
Dr Eike Hermann Mueller
Research Associate (PostDoc)

Department of Mathematical Sciences
University of Bath
Bath BA2 7AY, United Kingdom

+44 1225 38 5803
e.mueller@bath.ac.uk
http://people.bath.ac.uk/em459/