The passenger power window rolls down but not up?

It does the same thing when you roll down the window with the master switch as well. I took my door panel off to replace my speakers, and when I put the door panel back on is when the issue started. I put a multi meter to the passenger window switch and lose 6 volts when I try to roll the window up. Is that normal? I know the motor works because it will roll down, but you have to put a drill battery to it to roll it up. I need my window to roll down. What can I do to find out what the problem is? There are no fuse diagrams that say power window and I don't know how to check the relay. Please help me.