Dear Lawrence, When I run the attached file TOC2.py on a remote machine with a firedrake version updated from one week ago, I get the error that I sent to you in my previous email. When I run it on my laptop with an old version of Firedrake, it runs for a while and then stops with the error: ... 230 231 232 [math-mc1096:02125] mca_sharedfp_sm_file_open: Error, unable to open file for mmap: /tmp/OMPIO_h_232.h5_-43450367_.sm Exception petsc4py.PETSc.Error: 76 in 'petsc4py.PETSc.Object.__dealloc__' ignored Traceback (most recent call last): File "TOC2.py", line 84, in <module> dumb_file_medium = DumbCheckpoint(save_path+"medium/h_"+str(i_dt), mode=FILE_READ) File "/Users/mmfg/firedrake/src/firedrake/firedrake/checkpointing.py", line 65, in __init__ self.new_file() File "/Users/mmfg/firedrake/src/firedrake/firedrake/checkpointing.py", line 144, in new_file comm=self.comm) File "PETSc/Viewer.pyx", line 365, in petsc4py.PETSc.ViewerHDF5.create (src/petsc4py.PETSc.c:76259) petsc4py.PETSc.Error: error code 76 [0] PetscObjectDestroy() line 69 in /private/var/folders/7h/wbj8xp7n3g5cfbr32ctcmwzcy3jf53/T/pip-yGXqPh-build/src/sys/objects/destroy.c [0] PetscViewerDestroy() line 100 in /private/var/folders/7h/wbj8xp7n3g5cfbr32ctcmwzcy3jf53/T/pip-yGXqPh-build/src/sys/classes/viewer/interface/view.c [0] PetscViewerDestroy_HDF5() line 49 in /private/var/folders/7h/wbj8xp7n3g5cfbr32ctcmwzcy3jf53/T/pip-yGXqPh-build/src/sys/classes/viewer/impls/hdf5/hdf5v.c [0] PetscViewerFileClose_HDF5() line 39 in /private/var/folders/7h/wbj8xp7n3g5cfbr32ctcmwzcy3jf53/T/pip-yGXqPh-build/src/sys/classes/viewer/impls/hdf5/hdf5v.c [0] Error in external library [0] Error in HDF5 call H5Fclose() Status -1 ________________________________ De : firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> de la part de Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk> Envoyé : jeudi 8 février 2018 10:06 À : firedrake@imperial.ac.uk Objet : Re: [firedrake] .at() PETSC error Dear Floriane, On 08/02/18 10:03, Floriane Gidel [RPG] wrote:

Dear all,

I try to extract the values of a function h, at some specific positions with the function .at() .

I define a coarse mesh with

Nx_fine = int(Lx/0.4) + 1

Ny_fine = int(Ly/0.4) + 1

coarse_mesh = RectangleMesh(Nx_coarse,Ny_coarse,Lx,Ly,quadrilateral=True)

coarse_V = FunctionSpace(coarse_mesh, "CG", 1)

coarse_h = Function(coarse_V, name="h")

V_vec_c = VectorFunctionSpace(coarse_mesh, "CG", 1)

coarse_points = Function(V_vec_c , name="coarse_points")

coarse_points.interpolate(SpatialCoordinate(coarse_mesh))

Then I define a finer mesh with

Nx_fine = int(Lx/0.1) + 1

Ny_fine = int(Ly/0.1) + 1

fine_mesh = RectangleMesh(Nx_fine,Ny_fine,Lx,Ly,quadrilateral=True)

fine_V = FunctionSpace(fine_mesh, "CG", 1)

fine_h = Function(fine_V, name="h")

Then I load an h5 function into fine_h (defined on the same function space) with

dumb_file_fine = DumbCheckpoint(save_path+"fine/h_0", mode=FILE_READ)

dumb_file_fine.load(fine_h)

dumb_file_fine.close()

But then, when I try to extract the values of fine_h at the coordinates of coarse_mesh with

coarse_data_fine = fine_h.at(coarse_points.dat.data_ro)

I get this error:

[0]PETSC ERROR: ------------------------------------------------------------------------

[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range

This says that somewhere (presumably in the interpolation code), there was an out of bounds memory access (reading off the end of an array or similar). I tried to reproduce this, by didn't manage to. Can you post a complete code that fails? Thanks, Lawrence _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake

Dear Lawrence, Thanks. The firedrake status on the remote machine is: Firedrake Configuration: package_manager: False minimal_petsc: False mpicc: mpicc disable_ssh: True show_petsc_configure_options: False adjoint: False slepc: False slope: False packages: [] honour_pythonpath: True petsc_int_type: int32 cache_dir: /home/ufaserv1_i/mmfg/firedrake/firedrake/.cache Additions: None Environment: PYTHONPATH: None PETSC_ARCH: None PETSC_DIR: None Status of components: --------------------------------------------------------------------------- |Package |Branch |Revision |Modified | --------------------------------------------------------------------------- |COFFEE |master |5bb1b30 |False | |FInAT |master |ef12759 |False | |PyOP2 |master |8a918cf |False | |fiat |master |f2b2511 |False | |firedrake |master |a0a5206 |False | |h5py |firedrake |c1e4a81 |False | |libspatialindex |master |4768bf3 |True | |petsc |firedrake |3c589c1 |False | |petsc4py |firedrake |3f09f6d |False | |tsfc |master |d7da9dc |False | |ufl |master |50bbef8 |False | --------------------------------------------------------------------------- ________________________________ De : firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> de la part de Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk> Envoyé : jeudi 8 février 2018 10:53 À : firedrake@imperial.ac.uk Objet : Re: [firedrake] .at() PETSC error Hi Floriane, On 08/02/18 10:26, Floriane Gidel [RPG] wrote:

Dear Lawrence,

When I run the attached file TOC2.py on a remote machine with a firedrake version updated from one week ago, I get the error that I sent to you in my previous email. When I run it on my laptop with an old version of Firedrake, it runs for a while and then stops with the error:

Hmm, I just tried with an up-to-date Firedrake here (firedrake-status says): Firedrake Configuration: petsc_int_type: int32 mpicc: mpicc packages: [] disable_ssh: True honour_pythonpath: False show_petsc_configure_options: False adjoint: False cache_dir: /data/lmitche1/tmp/firedrake/.cache slope: False package_manager: False slepc: False minimal_petsc: False Additions: None Environment: PETSC_ARCH: arch-linux2-c-opt PYTHONPATH: None PETSC_DIR: /data/lmitche1/src/deps/petsc Status of components: --------------------------------------------------------------------------- |Package |Branch |Revision |Modified | --------------------------------------------------------------------------- |COFFEE |master |5bb1b30 |False | |FInAT |master |ef12759 |False | |PyOP2 |master |8a918cf |False | |fiat |master |f2b2511 |False | |firedrake |master |d53aaa8 |False | |h5py |firedrake |c1e4a81 |False | |islpy |master |bc69e45 |False | |libspatialindex |master |4768bf3 |True | |petsc4py |firedrake |3f09f6d |False | |tsfc |master |d7da9dc |False | |ufl |master |50bbef8 |False | --------------------------------------------------------------------------- And I get no errors. I don't think much has changed recently, but can you please post the output of firedrake-status just to see if there are any differences. ...

[math-mc1096:02125] mca_sharedfp_sm_file_open: Error, unable to open file for mmap: /tmp/OMPIO_h_232.h5_-43450367_.sm

Exception petsc4py.PETSc.Error: 76 in 'petsc4py.PETSc.Object.__dealloc__' ignored

Traceback (most recent call last):

File "TOC2.py", line 84, in <module>

dumb_file_medium = DumbCheckpoint(save_path+"medium/h_"+str(i_dt), mode=FILE_READ)

File "/Users/mmfg/firedrake/src/firedrake/firedrake/checkpointing.py", line 65, in __init__

self.new_file()

File "/Users/mmfg/firedrake/src/firedrake/firedrake/checkpointing.py", line 144, in new_file

comm=self.comm)

File "PETSc/Viewer.pyx", line 365, in petsc4py.PETSc.ViewerHDF5.create (src/petsc4py.PETSc.c:76259)

petsc4py.PETSc.Error: error code 76

[0] PetscObjectDestroy() line 69 in /private/var/folders/7h/wbj8xp7n3g5cfbr32ctcmwzcy3jf53/T/pip-yGXqPh-build/src/sys/objects/destroy.c

[0] PetscViewerDestroy() line 100 in /private/var/folders/7h/wbj8xp7n3g5cfbr32ctcmwzcy3jf53/T/pip-yGXqPh-build/src/sys/classes/viewer/interface/view.c

[0] PetscViewerDestroy_HDF5() line 49 in /private/var/folders/7h/wbj8xp7n3g5cfbr32ctcmwzcy3jf53/T/pip-yGXqPh-build/src/sys/classes/viewer/impls/hdf5/hdf5v.c

[0] PetscViewerFileClose_HDF5() line 39 in /private/var/folders/7h/wbj8xp7n3g5cfbr32ctcmwzcy3jf53/T/pip-yGXqPh-build/src/sys/classes/viewer/impls/hdf5/hdf5v.c

I think this is somehow an indication that some files were not correctly closed. I tried on my Mac laptop (also with the same firedrake indicated above), and didn't see any problems. A bit of a mystery! Lawrence _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake

Hi Lawrence, Thanks for the guidance. You'll find the output attached. Floriane ________________________________ De : firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> de la part de Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk> Envoyé : jeudi 8 février 2018 11:06 À : firedrake@imperial.ac.uk Objet : Re: [firedrake] .at() PETSC error On 08/02/18 10:57, Floriane Gidel [RPG] wrote:

Dear Lawrence,

Thanks. The firedrake status on the remote machine is:

OK, so the only differences are in firedrake itself (and those changes only update the documentation). To find out what's going on, we'll need to see a C level backtrace. Can you please run: $ gdb --args python TOC2.py This will start python running in a C debugger. You should see something like: ... For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from python...(no debugging symbols found)...done. (gdb) The "(gdb) " line is a prompt. We want to stop when we hit this error, so type: (gdb) break PetscError Function "PetscError" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (PetscError) pending. Say "y" to the prompt. Now we run the program: (gdb) run Eventually you will hit an error (I can't show you this because I don't see it!). At that point type: (gdb) backtrace full And please paste all the output. This will allow us to see where this out of bounds memory access is coming from (hopefully!) Cheers, Lawrence _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake

Hi Lawrence, with $ firedrake-clean $ export PYOP2_DEBUG=1 $ gdb --args python TOC2.py (gdb) break PetsError (gdb) run I have no error anymore. So it seems that it fixed the issue! Should I try to run it with $ python TOC2.py to make sure that it's fixed ? What was the issue then? Thanks, Floriane ________________________________ De : firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> de la part de Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk> Envoyé : jeudi 8 février 2018 11:27 À : firedrake@imperial.ac.uk Objet : Re: [firedrake] .at() PETSC error On 08/02/18 11:17, Floriane Gidel [RPG] wrote:

Hi Lawrence,

Thanks for the guidance. You'll find the output attached.

OK, that, unfortunately, was not very enlightening. We see that the segfault happens in auto-generated code, but can't see the problems. Let's try running the compiled code in debug mode. Do: $ firedrake-clean $ export PYOP2_DEBUG=1 $ gdb --args python TOC2.py And follow the instructions as before. Thanks, Lawrence _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake

It doesn't work outside the debug mode... I still get the same error: (firedrake) [mmfg@login1.arc3 3D_code]$ firedrake-clean Removing cached TSFC kernels from /home/ufaserv1_i/mmfg/firedrake/firedrake/.cache/tsfc Removing cached PyOP2 code from /home/ufaserv1_i/mmfg/firedrake/firedrake/.cache/pyop2 (firedrake) [mmfg@login1.arc3 3D_code]$ unset PYOP2_DEBUG (firedrake) [mmfg@login1.arc3 3D_code]$ python TOC2.py -------------------------------------------------------------------------- A process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. The process that invoked fork was: Local host: [[32797,1],0] (PID 77567) If you are *absolutely sure* that your application will successfully and correctly survive a call to fork(), you may disable this warning by setting the mpi_warn_on_fork MCA parameter to 0. -------------------------------------------------------------------------- 0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- ________________________________ De : firedrake-bounces@imperial.ac.uk <firedrake-bounces@imperial.ac.uk> de la part de Lawrence Mitchell <lawrence.mitchell@imperial.ac.uk> Envoyé : jeudi 8 février 2018 11:48 À : firedrake@imperial.ac.uk Objet : Re: [firedrake] .at() PETSC error On 08/02/18 11:38, Floriane Gidel [RPG] wrote:

Hi Lawrence,

with

$ firedrake-clean $ export PYOP2_DEBUG=1 $ gdb --args python TOC2.py

(gdb) break PetsError

(gdb) run

I have no error anymore. So it seems that it fixed the issue! Should I try to run it with

$ python TOC2.py

to make sure that it's fixed ? What was the issue then?

So it might be that by compiling in debug mode, a bug no longer appears. But let's try in non-debug mode: so do: $ firedrake-clean $ unset PYOP2_DEBUG $ python TOC2.py And see if you get an error. If not, it's possible just that somehow some compiled code was wrongly cached? Lawrence _______________________________________________ firedrake mailing list firedrake@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/firedrake