KorvaJoin the waitlist
Science

Shared Data Is the Only Clue a Study Can Be Rerun

Caveats, citation counts, and other markers failed, and AI trained on human bets could not pick the winners.

AI image
science.org · Across the social sciences, half of research doesn’t replicate | Science | AAAS

The only clue that predicted whether a social science study could be reproduced was whether its authors shared their data and computer code.

That finding comes from a large effort to find a quick credibility check for research. The team tested more than a dozen possible markers, including whether a paper reported caveats and how often it had been cited. No marker reliably predicted replication on new data.

Only about a third of the studies examined for reproducibility had posted their data and code for others to use, and those reproduced at a much higher rate. Data sharing predicted only that the same analysis could be rerun to the same numbers, not that new data would give the same answer.

Even artificial intelligence failed the replication test. Models trained to copy accurate human bets on which studies would replicate could not pick the winners. Brian Nosek, the psychologist who co-led the effort and directs the Center for Open Science, says much more evidence is needed before any scalable confidence score can be trusted.

Questions people ask

What does reproducibility mean in this context?

Reproducibility means taking the original data and computer code and rerunning the analysis to see if you get the same numbers. It is different from replication, which collects fresh data to test the same question, and from robustness, which tries a different method on the same data.

Why would sharing data make a study more likely to hold up?

Shared data and code let others check the work, catch coding errors, and see exactly how results were calculated. Researchers who share readily may also keep cleaner records and follow more careful methods, though sharing alone does not prove a finding is true.

Could a newer AI eventually predict which studies replicate?

Not yet, but researchers are still trying. A follow-up project run by the Center for Open Science is using a cash-prize competition to test newer forms of AI, and some entries have performed better than the earlier models.

More in Science