1988 Toyota Camry intermittent starting problem

This car has 180,000 miles and frequently will not engage the starter when you turn the key. It started about a year ago, every once in a great while, and has now increased to about 75% of the time. The starter is fairly new. Also, if you directly connect the starter to the battery it works every time, which makes me think the starter is fine. For this reason, I have attached a starter button in this manner for when the key doesn't work. I also just replaced the ignition switch and it has not solved the problem. The Toyota dealer told me this car (auto transmission) does not have a starter relay. Also, if you start it with the wired-in button and immediately shut it off, the key will work. The last thing I'm thinking about trying is the neutral start switch, but it has never made a difference whether the car is in park or neutral while the problem was just beginning. If you think this is the problem, how do I get at it to adjust or replace it?

This is the 2wd model with 4-cylinder and auto transmission.