Hi Kurt, Your use of AllReduce seems to be correct, but you have to guarantee that all processes call it because AllReduce is a collective operation. This locking usually occurs when there is a mismatch in the mpi calls, causing some processes to wait forever for a corresponding call in other ranks. Considering you are working with boundaries, it is easy to get this wrong, since in general only some of the partitions will contain the boundary. You can get around this problem by forcing the other ranks to perform dummy communications. For example, your code could look like flowrate = 0.0; if (bnd) { flowrate = BndCondExp->VectorFlux(bndVelocity); m_comm->AllReduce(flowrate, LibUtilities::ReduceSum); (do other things) } else { m_comm->AllReduce(flowrate, LibUtilities::ReduceSum); } Cheers, Douglas 2017-08-17 16:31 GMT-03:00 Kurt Sansom <kayarre@gmail.com>:

Hi community, I have implemented a new boundary condition and I referenced the implementation of the feature/flowrate branch to evaluate the flow rate on a boundary.

There were a couple places where m_comm->AllReduce was used and I emulated the form in solvers/IncNavierStokesSolver/EquationSystems/ IncNavierStokes.cpp

that look like the following:

m_RCParams[fldid][bndid]->m_FlowRateArea = BndCondExp->Integral(inArea); m_comm->AllReduce(m_RCParams[fldid][bndid]->m_FlowRateArea, LibUtilities::ReduceMax);

and

flowrate = BndCondExp->VectorFlux(bndVelocity); m_comm->AllReduce(flowrate, LibUtilities::ReduceSum);

but using AllReduce causes the solver to lock/freeze, should I be doing something like if(m_comm.GetRank() == 0)?

I can remove the AllReduce, but then I am repeating the calculation on each core correct? is there a best/better way to do this in nektar++?

Thanks for all your help and support!

-- Kurt Sansom Multiphase & Cardiovascular Flow Lab University of Washington

_______________________________________________ Nektar-users mailing list Nektar-users@imperial.ac.uk https://mailman.ic.ac.uk/mailman/listinfo/nektar-users