IMO, the safety plunger will cause the least amount of grit...prolly undetectable in many cases. Remember, it's a very small, weak spring that keeps it doing its job, so how much grit can it produce? Polishing it does, however, complete the entire trigger parts polish. The key spot (IMO, also) is where the trigger bar pushes on the plunger. A rough edge on the plunger or trigger bar, or both, can push the plunger in a way that it rides hard on one side of the plunger channel. That still won't produce much grit, but may be enough to feel. I personally sanded the top of the OEM plunger with 1500 wet dry paper just to round that 'edge' a little as well as the top finishing with 2000 wet dry. Also did the sides as well as the inside of the channel with the 2000. I use wet dry to remove the burrs which metal polish cannot always do, and when oiled, feels like rubbing two pieces of greasy glass together.
Replacing the plunger, as some do, is absolutely unnecessary...just clean up the OEM plunger.