Unless you shoot a wide variety of loads. My daughter likes to shoot softer target loads while I prefer hotter defense loads. The buffer is just extra insurance that no long term damage will occur.
A good many IDPA/USPSA shooters run springs as light as 10-12 pounds with tens of thousands of major power rounds fired annually. One local guy runs a 12-pound spring, and he installs a new one whenever he starts noticing sluggish return to battery. Last time I talked to him, he said his spring was probably due for a change, since it had logged close to 90,000 rounds. It looked rough, but it worked.
The slide just doesn't hit the frame all that hard, and everybody worries too much about the frame. It's the slide that catches all the hell. The slide and barrel assembly is the "gun." The frame is essentially the gun mount.