Really interesting! Thanks for sending this! It is only through such natural experiments that we can know the actual effect (or absence of effect). Would you have a preprint or an early version of the manuscript that you could send me? I don’t have access to it.
I thank my collaborators Valentin Thouzeau and Nicolas Baumard. And we thank Quentin Borredon, Robin Fraynet, Marius Mercier, Charlotte Touzeau, for their valuable feedback on this manuscript, and all the participants of the “Automatic Cultural Annotation” FRESH pre-conference workshop.
We would be more than happy to have feedback on this method from people who would try and use it in their research, or already are. Don’t hesitate to reach out to us for questions or remarks.
We are currently using this method in various projects and still refining the process (to enhance the quality of the automatic annotation and the reproducibility of the analyses). It is possible that we never end refining it!
Finally, we explain how this methodology could advance the empirical study of human cultures in radical ways, notably through the facilitation of comparative and dimensional approaches.
We believe it is now through empirical research conducted by experts in specific scientific fields that the true potential and limitations of LLMs in these fields can be effectively evaluated.
We propose a simple step-by-step method using R and GPT API, just like a tutorial. In Discussion, we review some studies from the extensive body of empirical research demonstrating the capabilities of GPT in zero-shot knowledge intensive tasks.
So we present the methodology we have been using and refining, with some advices we thought could be helpful for instance for prompt engineering or validity checks (as this method should always be rigorously tested).
Compared to other methods (participatory science, bag of words) we propose that LLMs have great advantages including cost-effectiveness (allowing annotation at scale) and uniformity (enhancing consistency and thus comparability).