2002 Other Chrysler Models fan switch or blower?

My 2002 voyagers blower fan works when it wants to. I can have it on any setting and no blower, the heater works because I feel the heat come through if i'm on the highway and I keep the vent/recirculate switch off to allow the forced air in from the vent only.
The blower will start occasionally while driving, work normal a few days, then quit again. Its my guess that its either a bad contact either on switch or blower fan, or the brushes on the blower motor are going.
is this a frequent problem and if so what is the most common reason or defective part (in most cases).

also is this a definite wallet buster usually or is this something I could replace?