Why won't my car start if it is hot outside?

My car won't start after sitting in the sun for a few hours. Every morning my car starts no problem. Runs perfectly when it does. It has driven from northern California to Las Vegas, San Diego and LA and back without overheating once. But on days that I drive to work and park the car at the end of my shift the car won't start and has taken various tines anywhere from 45 minutes to 4 hours to start. On occasion I have gotten frustrated and left the car to come back after sun down and it starts on the first try. I have had a few men say to me it sounds like a car with no gas but there has always been gas in it when it does this. So the first thought was the fuel filter changed that and it was still doing it. Second thought was the fuel pump changed that and the filter again still doing it. Checked all the fuses which also seem to be fine. I just don't know what it could be. And it is getting extremely frustrating. I don't always have an extra hour or more to sit and wait for the car to start and I can't always find a shaded place to park. Which by the way on cool days it doesn't do this and if it is in a very shaded area it doesn't do it then either. Please help this is a 2003 Oldsmobile aleo and I am just stuck literally and figuratively.