Outside air circulation is blowing warmer air than outside?

If I keep my A/C off, and set mode for outside air circulation, I expect the air from blower to be as cool as outside air since it is supposed to blow only outside air, but I feel it is warmer than outside. I want to be able to save fuel keeping A/C off if the outside air temperature is pleasant, but I have to necessarily keep A/C on in order to get cool air when outside air is cool already.