Intermittent not starting and shutting off issue

I can go out to start my car and turn the key and get absolutely nothing; no lights, no sound, it doesn't even try to turn over. I can play with the ignition and the column wiggle it and jiggle it and move it around, turn the key over and suddenly it fires up like it has no problem at all. I can pull up to a light be waiting for it to turn so I can go and the car will just suddenly it shuts off completely; no lights, no nothing. I sit there, I play with it jiggle it mess with it and it finally will turn over and start like it never had a problem and drive normally. I keep having the same issue. It's not the battery, brand new battery. I don't know what to do.