If I am elected president I will ban the phrase "there is no evidence." You will now need to say either "we have evidence of a lack of subtantial effect" or "we have no evidence to inform this question one way or another." Please Venmo me your campaign donations thx

Yep. Money where my mouth is: for the Registered Revisions project, the primary outcomes of the meta-trial are time-to-event-based process outcomes. Research outcomes are secondary/tertiary. www.tandfonline.com/doi/full/10....

But I think the list of Q's is missing one that I think is comparably important as (maybe even supercedes?) the research outcomes: process outcomes. Things like timelines, workload, author/editor/consumer satisfaction etc are suuuuper important here. Bonus: they are very measurable.

For what it's worth I am super pluralistic about #1. I've preregistered "we are just playing with things right now, none of this data is worth using inferentially" to full strict confirmatory stuff (and one I'll show later that's WILDLY beyond even the strictest current prereg practice).

Pre/post (and diff-in-diff and similar) is gonna be pretty sketchy since there is SO much happening that effects are likely to be near impossible to disentangle. And that's even if we had outcomes measured and had records of uncompleted projects etc. But I am on the hunt for those designs.

To be clear, I don't love replicability as the primary target. But if that were the target, replicability requires replication, so we're talking many years long projects. We'll see the results of at least one of those projects soon-ish, but don't expect strong evidence from that either.

Importantly though, there is a LOT of opportunity currently being left on the table. We can absolutely get to a better frontier and get better evidence, implementation (if appropriate), and outcomes much faster. The biggest lacks are creativity in study design and (of course) resources.

What we are ultimately left with is kinda basic in a lot of ways: Hedge our bets. Try things. Evaluate the things we are trying. Do our best to understand what we can and can't know with what we tried. Discuss discuss discuss.

"Do nothing until we have good evidence" is a self-fulfilling status-quo prophecy. "Charge ahead all-in without any evidence" clearly leads to bad outcomes. Theory is SUPER helpful, but isn't a 1:1 evidence substitute. So we have to strike a balance for which there isn't an obvious solution.

I have WAY too much COI to talk about the retracted preregistration paper, but suffice to say that even without the false statements about its own preregistration, it wasn't particularly informative on the impact of preregistration. So what are we to make of all this? What do we actually DO?