02 Cavalier sometimes will not start but pump is on

My car will run perfectly fine but sometimes will not start. I have replaced the fuel filter, and I can hear the fuel pump starting up, but the car will not turn over. It cranks but wont start. Then, twice, I had my car towed to the dealer and it started up when the mechanic got in. It has had hard starts where it takes a bit of cranking to actually start it but twice now, it would not start at all until I had it towed. This has happened only since I replaced the fuel filter (not starting), but the hard starts have been happening for a while now. Has anyone had this problem before? I don't want to replace the fuel pump if it is something else.