Hi Zhen,
Are the inlets to your domain cylindrical? VMTK has a widget that allows you to add cylindrical flow extensions. This would allow you have a more precise inlet, and womersley as implemented is only valid for a cylinder.
If there is a analytical solution to womersley of an arbitrary inlet crossection please let me know.
One thing to consider is that that analytical solution depends on the radius of the vessel, make sure R is the same units as the domain, and if the radius of the mesh inlet exceeds the radius, R for the Womersley solution, then failures would be expected.
I do similar simulations in fluent and have found that rounding to single precision in the womersley parameters avoids this issue.
~Kurt