Perhaps try a good leather gun belt? My problem isn't drawing the pistol. My holster keeps it easy to draw, no matter what. It's the fact I'm a little overweight (5' 8" and 165 pounds) and I have no hips.
So if I don't chinch it down tight enough I have to keep pulling my pants up. If it's too tight the pressure gets to my waist. I refuse to wear suspenders, although I'm sure that would solve the problem.