2003 Pontiac Grand Am Hard cold start

Car usually doesn't start in the mornings and if it does it takes 5 or more seconds of cranking. The weird part is later in the day when I get off work and try to start the car it always starts right up and runs fine. It also starts up fine as long as I don't let it sit overnight or for an extended period of time.

I know there is a security issue with these cars but I don't think that's it because the light is not on or flashing. I hear the fuel pump come on every time and I've tried turning it off and on a few times to make sure its primed encase the fuel pump has a leaky check valve. There are no leaks in any lines or in the fuel pressure regulator and I also put a new fuel pump relay in. I'm at a loss on what it could be. When it starts it runs fine. There is no stalling, hesitation ect. The previous owner said he removed remote start from the car so I'm wondering if that could have something to do with it? Bad wire or something?