It's about the "time line". You have to be able to stay alive until society gells again, the crisis ends or enough of your buds conclude there's safety in numbers.
Depending on the type of incident, you may indeed be able to fend off low level turds by mere brandishing with the wife and kids. As a crisis develops the risk vs. reward process comes into play...do the crooks think it's worth risking their lives? Initially they won't be aware of the fact that they are facing a few people willing to put up a defense. That will be the time you could get exploited...while you are just a gaggle.
Most people don't have the wherewithal to pull off squad level infantry tactics...they lack the will, equip, training, and will not want to do harm. If you can rally a few folks, great, but most families will have kids, elderly, slacker, etc.