I have wondered the same thing, especially for hiking in the wintertime (folks in heavy coats, not the dreaded "hiking bear"). However, there are some good, common-sense replies in this thread that have helped me realize that I am probably overthinking this one, like I do with everything.
For me it will be 125gr GD in 357 sig or 165gr GDs in .40 S&W year-round, until I overthink something else