I don't see any major downside, imagine the capabilities to learn new topics and also reference the proper manuscripts. For example, I sometimes unintentional miss important references just because it is to much. I will love to solve that.

It gets at my argument that chatgpt may not be perfect, but it just needs to be as good as an average human, and I know the latter misses references or has reference-bias, so why not. And the energy cost is exponentially lower with unconstrained topic foci.