Fuel pump keeps running

Turn the key on and see if the pump relay clicks on. It is possible that the old pump was drawing enough current that it fused the relay. If you do not feel the relay click on replace it and see if the problem is solved.

OK, that sounds more like the pump is coming on for the initial prime and then shutting off. What was the reason the pump was replaced?

That would fit the prime only supposition. The system normally turns the pump on for a couple seconds, then when the PCM reads a higher than cranking rpm signal, meaning the engine started on the prime fuel it turns on the pump and the engine runs. In your case it sounds like the prime signal is okay but it is not turning the pump on. That suggests that the PCM is not getting the rpm signal from the crank position sensor. Or it is getting the signal and not responding properly.
A test you can use would be to jumper the relay so that the fuel pump stays on and see if the engine runs. That can be done by applying twelve volts to terminal 13 on the diagnostic connector U-01 under the hood (this is not the OBDII connection under the dash).

If it runs you will need to get a scan tool that can read live data (even a low end tool can usually do that) and check to see if there is an rpm signal. If it is missing it is likely the crank sensor.