Pontiac door handle

Looking for a little info. I have a 1994 pontiac grand prix 4 door v6. I was sitting in my car the other day, opened up my door and shut it. About two minutes later I went to open up my door, and the handle pulled out by it seems like it's not grabbing on to anything. Do you think if I take the door panel off, whatever it latches on to just fell off, or do you think whatever connects to it broke?