If it comes on at times and not others, chances are either the motor itself is going bad, or there is a problem with the blower motor resister.
What I would recommend is to turn the fan on. When it isn't working, check to see if the motor is getting power. If it is, then either there is a bad ground to the motor, or the motor is bad. IF there is no power, then I would move to the relay and check for power there.