Deterministic versus indeterministic is a red herring. There are lots of indeterministic algorithms in statistics (like MCMC). The question is reliability at the task it's been handed. LLMs are not reliable at addition, so they shouldn't be used for that.

But they might be reliable at other tasks. We should not presume they are or are not, we should figure it out by testing. If they are reliable, then we should use them. If not, we should not.