2001 Ford F-250 electric windows intermittently work.

The power windows, this a club cab pickup, work intermittently. I have replaced the drivers side dual switch and the drivers side motor. Now when I try to roll down the passenger window, using the drivers side switch it will operate fine in one direction then if you try to go the opposite way it will not work unless I wait for the window relay, located under the dash, to click and then it will work fine in that direction. The passenger side switch does the same thing. The drivers side window will go up, but not down. I have tried switching relays and the same thing happens.