2001 Volvo S40 Air Conditioner does not blow all the time

My A/C does not blow out all the time. If I start the car and it doesn't start correctly, then I turn the key all the way off and restart the car again the A/C blower will start working, but not each time I do that. When I simply start the car on the first try with no problems the A/C blower will never come on. But, for some reason if I turn the ignition a little wrong where the car doesn't start, and then I have to turn the key off and turn the ignition a 2nd time then the a/c blower will come on but not everytime that way either. Is there a switch in the ignition that connects to the A/C that is going bad? I tried turning the switch on the a/c off and on several times and the blower will never come on. I get a cold air flow if I'm on the interstate doing 60MPH and the sunroof is popped up.