I'd bet the single biggest reason for this is not having good (flexible, durable, simple) tools to dynamically adjust NPC speed. BioShock Infinite DID solve this for Elizabeth - while she was supposed to be following you, we also wanted her in sight as much as possible (ie in front of you). So...

If I'm in a situation where I can't make an action that's generally not enjoyable into something enjoyable, I'll try to work around that. Obviously in Infinite, you'd have to with Elizabeth. But in the game I'm playing now, they didn't have to do a walk and talk. It would've been better not to

So we used the NPC pathfinding system to build a route from the player to the next objective. That path, near/in sight of the player, was her ideal location when out of combat. We constrained her speed most of the time, but would teleport her when off screen & let her sprint into place.