1994 Jeep Cherokee Low oil pressure

Hi guys. Ok I am a mechanic myself but seem to be stumped on this one. My inline 6 4.0L was having low oil pressure and bearing noise. I replaced all the top end and bottom end including oil pump. I did notice though that there is a hole behind the cam sprocket that you can see a push rod. First thing is would this hole there about a nickel size cause my low oil pressure? The engine now has no noise since it is all new. Also I don't burn a drop of oild between oil changes and don't burn any antifreeze either. When I first start up it is sitting at 20-30 and than when it warms up it drops to 4 lbs. I am using 10W30 pennzoil. Any help on this would be appreciated as I don't want to drive it with the low oil and blow out my bearings. When you step on it the oil pressure goes up to 20 lbs or more. HELP! I am completely at a loss. Thanks in advance for any help here. Oh yeah I am using a mechanical gauge for the reading also.