2003 Toyota Sequoia Whistling Sound

There is a high pitched whistling sound coming from where the brake pedal goes thru the firewall. Does not occur when brakes are applied even slightly or when under acceleration and is louder if brake pedal is pulled back with your hand. Doesn't happen if the engine is cold and only starts after about 10 miles. Brakes work fine otherwise, no hard or soft pedal. I have been told it must be the brake booster but I think there may be a vacuum leak elsewhere. What do you think. Thanks