Intermittent grinding when clutch is released.

I have a grinding I feel when the clutch pedal is pressed. It is occurs more when the vehicle is cold. I feel it in pedal and the shifter, but more so in the pedal. You can push down the pedal and hold it with no grinding but as soon you move the shifter from neutral (between 3rd and 4th gear) to between 1st and 2nd it will start to grind. As cold it will do it 75% of the time and when at operating temp it will only be 25% of the time. I'm thinking its the clutch release bearing, but wouldn't it be all time?