Are you proficient with either?
You can pick up a very good quality pump shotgun for far less than an AR-type rifle, which makes it a good starting point, but if you're not proficient with it, it's a handful.
Do you have a sporting clays course near you? Sporting clays are a super fun way to become proficient and comfortable handling a shotgun. Don't get a stockless pistol grip gun. They look badass, but you want to be able to shoulder a shotgun to control it. The only mod you might want is a light.
As for a rifle, there are many, many options that can get you started a lot cheaper and more practically than an AR. For home defense, a simple no-frills SKS and a handful of 10-round stripper clips will do the job just fine.
Don't modify the SKS in any way -- don't be tempted by the fancy stocks or detachable magazines. They all detract from the simple combat usefulness of the rifle. With practice you can load from 10-round stripper clips as fast as you can change a magazine.
Again, you'll want a range that lets you shoot rifle-caliber. The SKS is a very easy point-and-shoot rifle, very easy to maintain, and makes a good club or spear in a pinch.
The unmodified SKS, if it has a bayonet, doesn't have a decent place to put a light/laser. If the bayonet has been removed, or you remove it, you can mount a light/laser there.
Another choice for home defense is a pistol-caliber carbine, like a Kel-Tec Sub2000 or the infamous Hi-Point 9mm carbine (don't laugh, they're ugly but cheap, reliable and accurate). Pistol-caliber carbines are more controllable and pointable in a stress situation than a handgun. They won't have the over-penetration problems an AR or SKS might, and unlike a shotgun, recoil is almost non-existent. Loaded with quality hollowpoints, they will seriously discourage an intruder.
They are really easy to control and handle in close quarters. Stick a laser/light combo on and you have a very handy, easy-to-use home defense weapon. 9mm is also one of the cheapest of all rounds to shoot and most pistol ranges will allow you to shoot pistol-caliber carbines at them, which makes finding a practice range even easier.
I have access to all the options mentioned and the 9mm carbine is by far the simplest and easiest to wield.