Ignition problem

I have replaced the ignition lock cylinder, the ignition switch and the starter solenoid. The key turns to the start position but will not turn back. Consequently, the starter continues to run until I pull the battery cable. I cannot think of what else could be the problem.