2002 Honda Civic driver's side seat belt buckle

It is becoming increasingly difficult to latch the seat belt on the drivers side. It seems to get worse with time, you have to push it in several times 10-12 or more before it finally clicks and locks. Even then, on occassion it will come unlocked while you are driving down the road. I would like to replace it, but not sure what all is involved in the replacement. It looks like the seat needs to come out to access the buckle. I was wondering how difficult of a job this is. Is it possible that there is crumbs or dirt in the bottom, preventing it from full insertion and causing it to not lock?