Hi Matt, Are you sure? To me it looks like pulp is using subprocess to fork the linear program solver. Nothing to do with parallel. David On Thu, 30 Mar 2017 at 14:30 Matthew Knepley <knepley@gmail.com> wrote:
What is going on here? It looks like you are using subprocess. Why would you do that on a cluster rather than MPI?
Matt
---------- Forwarded message ---------- From: *Buesing, Henrik* <HBuesing@eonerc.rwth-aachen.de> Date: Thu, Mar 30, 2017 at 8:25 AM Subject: AW: [petsc-maint] Weak scaling test: Fieldsplit questions To: Matthew Knepley <knepley@gmail.com> Cc: Barry Smith <bsmith@mcs.anl.gov>, Hong <hzhang@mcs.anl.gov>, " petsc-maint@mcs.anl.gov" <petsc-maint@mcs.anl.gov>
In my opinion, there is some kind of race condition in Firedrake when running on more than one node. Thus, until this is fixed it is very unlikely for me to get the 64 cores case running.
Hmm, we are running Firedrake in parallel with no problems here. What is the error?
*[Buesing, Henrik] *See [1] for the error message and the attached three logs (for the 32 core case this was a 2/5 running and 3/5 crashing ).
This is just for running the compiled code. During the compile stage I had problems, too. What I did is the following: 1) Run Firedrake on 1 node (this works). Now all the *.so files are in place. 2) Run Firedrake on more than one node. This crashes more often the more processes I use.
I’m guessing for a race condition, because on 17 cores (1 node + 1 core) my problem runs fine. On 32 cores it sometimes runs. And on 64 cores it, up to now, never runs.
But if you are not having these problems, and if the provided code reproduces the MatCreateSubMats problem, then you can do tests on your own. Well, a lot of ifs, but better than nothing.
Thank you! Henrik
[1]
Traceback (most recent call last):
File "/work/hb111949/Firedrake/twophase/2pDrake/2pinjection.py", line 228, in <module>
solver = NonlinearVariationalSolver(problem,options_prefix="")
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/variational_solver.py", line 156, in __init__
pre_function_callback=pre_f_callback)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/solving_utils.py", line 260, in __init__
form_compiler_parameters=fcp)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/assemble.py", line 143, in create_assembly_callable
collect_loops=True)
File "<decorator-gen-279>", line 2, in _assemble
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/utils.py", line 62, in wrapper
return f(*args, **kwargs)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/assemble.py", line 192, in _assemble
kernels = tsfc_interface.compile_form(f, "form", parameters=form_compiler_parameters, inverse=inverse)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/tsfc_interface.py", line 193, in compile_form
number_map).kernels
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", line 200, in __new__
obj = make_obj()
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", line 190, in make_obj
obj.__init__(*args, **kwargs)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/firedrake/firedrake/tsfc_interface.py", line 121, in __init__
kernels.append(KernelInfo(kernel=Kernel(ast, ast.name, opts=opts),
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", line 200, in __new__
obj = make_obj()
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/caching.py", line 190, in make_obj
obj.__init__(*args, **kwargs)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/base.py", line 3843, in __init__
self._code = self._ast_to_c(self._ast, opts)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/PyOP2/pyop2/sequential.py", line 73, in _ast_to_c
ast_handler.plan_cpu(self._opts)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/COFFEE/coffee/plan.py", line 121, in plan_cpu
loop_opt.rewrite(rewrite)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/COFFEE/coffee/optimizer.py", line 117, in rewrite
ew.sharing_graph_rewrite()
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/src/COFFEE/coffee/rewriter.py", line 619, in sharing_graph_rewrite
prob.solve(ilp.GLPK(msg=0))
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/lib/python2.7/site-packages/pulp/pulp.py", line 1651, in solve
status = solver.actualSolve(self, **kwargs)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/lib/python2.7/site-packages/pulp/solvers.py", line 383, in actualSolve
rc = subprocess.call(proc, stdout = pipe, stderr = pipe)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/lib/python2.7/site-packages/subprocess32.py", line 578, in call
p = Popen(*popenargs, **kwargs)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/lib/python2.7/site-packages/subprocess32.py", line 825, in __init__
restore_signals, start_new_session)
File "/rwthfs/rz/cluster/work/hb111949/Firedrake/firedrake/lib/python2.7/site-packages/subprocess32.py", line 1574, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 14] Bad address
Thanks,
Matt
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- Dr David Ham Department of Mathematics Imperial College London