2003 BMW 325 starting problems when recently driven

My car will start easily when it has not been started for at least a couple of hours. But if I have driven it and turned it off, and come back to it within an hour or so, it requires about 5-10 seconds for it to crank before it finally starts. Do you know what the problem could be? The external temperature has no bearing on its starting, just the amount of time that it has set after being turned off before trying to start it again. It's very frustrating and embarrassing.