2002 Dodge Ram Radio is running my battery dead

Electrical problem
2002 Dodge Ram 4 cyl Two Wheel Drive 115000 miles

I have a 2002 Dodge Ram 4.7L with 115,000 miles. My battery has been weakening and/or is dead in the mornings or over period of time of not driving it. I have the Infinity Sound System and since I've removed the fuse for the "radio amp" it has not weakened or gone dead, narrowing the problem down to the radio and/or the infinity sound system. Any suggestion of what might be causing this problem? Thanks!