I'm sure the book mentioned covers this, but I'll take a shot. Let's start at the beginning of WWII.
Most countries were using rifles with full power .30-ish caliber bolt action rifles (English Enfield in .303, German Mauser in 8mm, US '03 Springfield in .30-06.) Long range accuracy was considered of key importance. The US took a half step toward higher firepower by going to the semi-auto Garand. We also took a further step with the M1/M2 Carbine in the lower power .30 Carbine round.
But what was discovered during WWII was that the vast majority of soldiers never took a shot longer than 400 yards, and that the long range capability of the full power rounds was wasted. And as tactics evolved, the slow firing long range bolt guns proved less than ideal. On the flip side, the full auto firepower of submachine guns was very useful in close quarters fights, but subguns had very limited effective ranges. What was needed was a gun with select fire capability that fired an intermediate power round (more powerful than the pistol caliber rounds used in subguns, but less powerful than the full power rifle round.)
The first gun to explicitly pull this together was the German StG44 (Sturmgewehr or in English, Assault Rifle, as named by Adolf Hitler.) It was smaller and lighter than the Mauser 98, it had full auto capability, and it fired a shorter, intermediate power 7.92mm Kurz round, which allowed it to be controllable in full auto fire. This was a much more effective infantry weapon than the bolt guns or pistol subguns, but came to late to affect the outcome of the war.
While not designed explicitly for the reasons the StG44 was, the select fire M2 carbine fits into pretty much the same mold.
Immediately after the war, the Russians came out with something very similar, that is very well known today, the AK47. This gun very much follows the pattern set by the StG44.
It took a while for the US to explicitly get with the program. We took an intermediate step with the M14. This was basically an updated, select fire M1 Garand. It didn't really get tested until we got involved in the Viet Nam war. There it proved too heavy, to cumbersome, not controllable under full auto, and the ammo load out carriable by a single soldier proved insufficient. Around this time Eugene Stoner designed the AR10, in full power .308. Given the ammo issue, he downscaled it to .223 Remington.
While this was essentially considered a varmint round at the time. Someone figured out that, given it's high velocity, if it was understabilized, it would tumble on impact and fragment, causing much more damage than a round of it's caliber should on a human sized target. And, since it was a much smaller cartridge than .308, a solder could carry a much larger round count. So that's basically how we got to the M16/5.56x45mm combo. This combination was controversial (maybe even still) and was not without it's teething problems. But in the end, it likely proved the correct move. This was acknowledged by the Soviets, when they went to their own small caliber, high velocity round in the AK74.