New preprint! How often do editors/reviewers actually compare preregistrations to the reporting in the article? Almost never. This figure sums up all of the results. Not good.

Very interesting! It's tough to assign responsibility. Do you think reviewers/editors should check, or authors directly report in the paper if changes have been made from preregistration? As we to some degree have to trust that data is not fabricated, shall we trust authors on this too?

Great work! Clearly a lot more to be done in this space

This is a valuable paper. I do wonder about the generalisability of findings from Plos, though. In my experience, less prestigious generalist journals like Plos tend not to have the highest quality peer review. Perhaps things look better at journals with stronger community endorsement?

Shouldnt the reviewer have latitude to decide if checking the preregistration is important or not?

Not obvious to me why it's the editor's job or the reviewer's job to check the preregistration. We don't expect these people to check the paper for fraud either.

Interesting stuff! Reminds me of this paper: doi.org/10.1371/jour...

If a paper has a straightforward analysis, doesn't drop any observations, etc then consulting prereg may not be especially informative. More informative if authors are, say, predicting a 3-way intx or dropping many observations. I'd like to see how often prereg isn't consulted in those latter cases.

Depressing

cc @jnfrltackett.bsky.social@brendannyhan.bsky.social, these are the data I mentioned a few weeks ago. Limited and preliminary, but discouraging.

I did so during my last year at JESP, taking me 15-30 extra minutes per article, but the results were eye-opening. Ideally publishers would peel off some of their whopping cost margins to staff journals properly for quality control, but I don't know what realistically is going to happen.