Why would the starter activate when the ignition is in the on position

Stay activated even after the key is removed? Car has been siting for about five years. Put in new battery and about two gallons of gas, hit the key and it sorta started but didn't. When I turned the key off the starter just kept on cranking until I disconnected the battery. I took the starter to a local parts store to have tested they said it passed, replaced ignition with one from junkyard and got the same results.