1996 Toyota Celica CALIPER BRAKET IS TOUCHING INNER SIDE OF

I have a 1996 Toyota Celica Gt. Recently I can hear a strange noise (like 2 metals grinding each other) coming from my left wheel while driving. When I turn left the noise get louder. I thought it may be brake pads or the rotor. So I replaced the rotors and the pads both. The sound was still there. Then I removed the wheel again and inspected the pads and the caliper. I noticed that the inner side of the rotor actually touching the caliper bracket (torque plate). When the car front is jacked up I can see there is only a 1mm space between the caliper bracket and the inner side of the rotor (The outer side of the rotor and the caliper bracket has about ½ an inch of space). It seems that under the load, while driving and turning left actually makes the inner rotor touch the caliper bracket. I also noticed that the some parts of the rotor brushes off the caliper bracket more. It just seems like the rotor is uneven also, but since the rotor is brand new, I suspect, may be the wheel bearing could be the culprit. I also installed a brand new ball joint. Could it be a bad caliper bracket also(torque plate)?