If you have come across this paper and it has influenced how you have approached generative AI in your workplace or research, can you send me a message (or reply here)? Thanks very much!
My paper with Oliver Hauser on generative AI was published in July and has just gotten over 25,000 downloads! Very proud of our work and the impact it has had. We have spoken with a number of companies and the paper has been cited in policy and company reports. www.science.org/doi/10.1126/...
Generative AI can enhance the creativity of short stories but may limit the variation in diverse outputs.
Thanks Joshua. Let me know if you have any feedback!
I suppose another way to answer is that the words, clauses, etc are fitted on past data but how they are then outputted in response to novel inputs (prompts) is not necessarily limited to past data.
If I understand, the answer is I am not sure. I suspect that either it does or it could be instructed to consider future uncertainty in its response. That may increase variance in responses but whether the variance arises from extrapolating beyond the past… I’m not sure.
I hope I did that thread properly in my previous post. Anybody have any tools to simplify that?
There are some additional details in the paper itself. We are really excited about these findings and we think there are multiple immediate and practical implications from the work. Please let us know if you have any questions!
So what are the takeaways? The perspective of generative AI can be useful when you average out the weirdness of any one LLM response. It is useful to treat generative AI, not just as one soundboard or assistant, but a crowd of opinion givers.
First, *individual* evaluations are quite problematic. LLMs (just like people) tend to be biased and inconsistent. Second, when we put together the evaluations from *multiple* LLMs and/or prompt approaches, generative AI evaluations look pretty similar to the strategy professors.
We have a set of 60 business models and we ask generative AI to rank the business models (based on a series of pairwise comparisons), and we also ask over 50 strategy professors to do the same. What do we find?