Transmission is acting funny

So, I'm driving on the interstate, tried to downshift (automatic) and I see a cloud of smoke in the rear view, and my check engine light comes on. So I pull over, pop the hood. It's smoking from the left side near the firewall, but I can't pinpoint the location. I also noticed the overflow tank had overflowed over, meaning to me, it's boiling, got too hot, but why is my question. Now, I wait for about 40 minutes to start it up again, and it tries to turn over, but it won't. I try 5-6 times, and still a no go. I try to start it in neutral. It turns over. So I put it in D, and it feels like I had a clutch and I tried to pull off in second. I drive home, a little over 100 miles to go from that point, doing around 60. I think it never went out of 3rd. Every time I stop, it stalls. Still doesn't start in park. But I made it home safely. So I'm hoping you guys might have a thought on if it's a problem I can fix myself, or I need to replace/rebuild the tranny. Thanks.