First suspect is a collapsing vacuum hose to the power booster. Next would be the check valve. If the power assist comes back when the transmission downshifts, that's because engine speed and vacuum picked up. That points to the check valve. It is supposed to let the booster store enough vacuum for two power-assisted stops if the engine stalls. That vacuum goes away when the check valve is leaking and engine vacuum is low.