Remember you have a loaded mag with a fully compressed spring and your jamming the mag up into the gun. Now the top bullet is pressing on the slide from underneath and trying to push that already compressed mag spring down even further.
Yes, it will take more effort. Just give a light rap on the bottom of the mag with your palm to seat it firmly.
Empty mags don't have the mag spring compressed and therefore are easy to insert. The follower just pushes lightly against the bottom of slide.
It will get easier as your mag springs wear in too.