Wont shift into gear

I have an 03 Saturn Vue - 4 cyl. Wth Manual transmission. Prior to yesterday it was running fine without any difficulties. During the day yesterday I noticed that it was a little hard to shift into gear a couple times. The next time I tried to change gears it was fine so I didnt worry about it and assumed that I wasnt pushing the clutch in enough or something along those lines. After driving around town for 3-4 hours I got on the interstate for about 30 miles. As I got off the interstate it refused to shift into any gear. It would move in neutral but would not all the gear to engage. I stopped and opened the hood I looked at the fluids and the transmission fluid was at the low mark but not empty. I replaced the cap and got into the car to try it one more time and it allowed me to change gears with no problem. So I drove it toward home. It was fine on the return trip until I got to the red light near my house and had to change gears at which time it refused to go into gear again. I did finally get the car home and added transmission fluid. This seemed to work and while it was a little hard to shift it was driveable. While adding transmission fluid I noticed that the fluid looks dirty and probably needs to be flushed. This morning the car was again refusing to go into gear. I am wondering if it could be the transmission master cylinder. Any suggestions are appreciated.