Lower control arm bushings worn out

My truck listed above has worn lower control arm bushings. They are cracked and allow some forward movement when the brakes are applied. Steering is tight and responsive. I have a clunking (dull sound, not metallic) sound when driving slowly on uneven surfaces or if steering quickly to the right or to the left. The sound tends to go away in colder weather. When prying the control arm with a pry bar, there is some movement, but it returns to its original position. Tires seen to be wearing okay. Am I doing harm by continuing to drive, such as hurting the lower and or upper ball joints?