2003 Chevy Silverado Differential Clunk

I have a clunk in my rear diff when I back up and shift to drive to go forward. If I back up and shift to nuetral then to drive instead I do not have a bang or clunk of any kind. And when I back up and shift to drive I am doing it very carefully and I am at a complete stop when shifting. The clunk is bad enough that the first time it happened I actually got out of the truck to see what I hit. Any ideas?
Thank you

Edit: This is in a 3/4 ton duramax