2001 Pontiac Grand Am Start up problem

My 2001 grand am 3.4 v6 has trouble starting up especially in the morning after sitting. I hear the fuel pump kicking on and once started it runs like a champ no hesitation or anything. What I have figured out is that when I turn the key to on for a few to five minutes it starts right up but if you just turn the key and try to start without waiting a minute or so it will just turn over but not fire. It will also start up if you use starting fluid. Im thinking it is the fuel pump but if the fuel filter needs changed would that cause it longer to get fuel to the engine to start and thats why when I turn key on for a minute it starts up fine?