It sounds like fuel pressure is bleeding down when the car sits. I'm not familiar with what it takes to turn your fuel pump on, but some cars, Chryslers in particular, will run the pump for one second when the ignition switch is turned on. They do that to insure fuel pressure is up and ready for starting the engine. If your car does something similar, turning the ignition switch to " run", then back off after a few sconds, then cranking the engine will make it start faster. That doesn't fix a problem, but it could help identify it.
Some cars turn the fuel pump on through an oil pressure switch on the engine. If your oil level is low, it can take a long time to build up enough pressure so the injectors can spray the fuel into the engine. That could also be identified by measuring the voltage on that oil pressure switch to see how long it takes to turn the pump on.
Fords use a goofy " inertia switch" to turn the pump off in the event of a crash. In those systems, the pump runs all the time the ignition switch is in the " run" position. If your car has a similar system, simply waiting for a few seconds with the ignition switch in the " run" position before cranking the engine will overcome the long crank time.
If any of these things makes the engine start faster, the cause of the loss of fuel pressure must still be found. There are three places to look. A leaking check valve in the fuel pump will not cause a problem in itself. A leaking pressure regulator can leak fuel back to the tank. That also is not harmful, but it can also leak through the vacuum hose and dump excessive fuel into the engine. That can lead to engine damage, but I doubt that is what's happening because your engine would not run very well.
The third, and more common cause of loss of pressure is a leaking injector. That can be harmful because the raw fuel can wash lubricating oil off the cylinder wall and dilute the engine oil. Lack of oil leads to very rapid cylinder and piston wear. Gas in the oil that doesn't burn off reduces the oil's ability to separate moving parts and greatly increases emissions when the fumes are pulled out and burned in the engine.