If we're talking quantitative summaries of data, humans are basically just selecting mean vs. median and handing things off to a deterministic algorithm which isn't a black box at all and can be tested for accuracy While LLMs are confidently unreliable at adding five-digit numbers together

Deterministic versus indeterministic is a red herring. There are lots of indeterministic algorithms in statistics (like MCMC). The question is reliability at the task it's been handed. LLMs are not reliable at addition, so they shouldn't be used for that.