Get involved with your HOA and provide some balance to their decision making. I'd guess that your fellow members may have had the same experience. In my HOA, they can't do anything until they notify the home owner when someone from the board verifies the situation exists. They try to take care of it internally. Usually it's about RV's being parked in driveways, but it still comes down to someone complaining, and that's rare during camping season..
We are told by our government NOT to judge Muslims by the actions of a few crazies. We are also told by this same government TO judge ALL gun owners by the actions of a few crazies.