Could be something wrong with the slide stop, maybe the spring is weak or kinked, or could be you're accidentally pushing the slide stop up while firing. Or the slide stop could've been installed wrong if you've ever taken it apart and put it back together.
Edit: If the slide stop spring is weak, the slide stop can 'pop' up into the notch in the slide with recoil.
With the pistol field stripped, hold the frame in one hand and gently pull the slide stop up. Don't use too much force or lift it too far, but it should snap back down toward the frame when you let go of it. If it's just kinda loose or doesn't wanna move properly you'll know that's your problem. It won't have very much tension on it so if the spring is weak it'll likely just flop around.
Hmm . . . Was this one particular mag or was this situation duplicated with multiple mags?
Suggestion . . .
1. Examine the slide stop/release assembly for damage.
2. Put one round (live or snap cap) into a mag and insert it into your G17. Rack the slide. If the slide locks itself at the rear/open position, examine to see what is activating the slide stop assembly into the active position.
If the slide stop is undamaged, only the mag follower, in an empty mag, should be able to activate the slide stop assembly.