This is what "worldbuilding" as a concept brings to the table as distinct from long-established literary concepts like "setting" or "background." Since this is what makes it unique, this is generally what its critics mean when they (we) have something to say about it.

This is why you can have books that are heavy on worldbuilding but provide almost no conception of milieu.

I always thought this distinction was too obvious to require making lol