Measure the amount of vacuum at the hose at the modulator.
It must be the same as engine vacuum off the intake manifold.
If there is any difference that must be corrected.
Even a small insignificant leak will throw a monkey wrench into the works.
By you description it sounds like a vacuum problem.
