Throughout the last decade, the so-called replication crisis has stimulated many researchers to conduct large-scale replication projects. With data from four of these projects, we computed probabilistic forecasts of the replication outcomes, which we then evaluated regarding discrimination, calibration and sharpness. A novel model, which can take into account both inflation and heterogeneity of effects, was used and predicted the effect estimate of the replication study with good performance in two of the four data sets. In the other two data sets, predictive performance was still substantially improved compared to the naive model which does not consider inflation and heterogeneity of effects. The results suggest that many of the estimates from the original studies were inflated, possibly caused by publication bias or questionable research practices, and also that some degree of heterogeneity between original and replication effects should be expected. Moreover, the results indicate that the use of statistical significance as the only criterion for replication success may be questionable, since from a predictive viewpoint, non-significant replication results are often compatible with significant results from the original study. The developed statistical methods as well as the data sets are available in the R package ReplicationSuccess.
","journal":null,"year":null,"id":590677,"datarank":0.2552340931839937,"base_score":1.6094379124341003,"endowment":1.6094379124341003,"self_citation_contribution":0.24141568686511508,"citation_network_contribution":0.013818406318878639,"self_endowment_contribution":0.24141568686511508,"citer_contribution":0.013818406318878639,"corpus_percentile":null,"corpus_rank":null,"citation_count":4,"citer_count":3,"citers_with_citation_signal":1,"citers_with_endowment":1,"datacite_reuse_total":0,"is_dataset":false,"is_dataset_confidence":null,"is_data_producer":false,"deposit_databanks":null,"is_oa":false,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":null,"fair_score":null,"fair_percentile":null,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":24177,"name":"Leonhard Held","orcid":"0000-0002-8686-5325","position":1,"is_corresponding":false},{"id":24178,"name":"Samuel Pawel","orcid":"0000-0003-2779-320X","position":0,"is_corresponding":false}],"reference_count":0,"raw_metadata":{"has_enrichment":true,"resolved":true,"title":"Probabilistic forecasting of replication studies","abstract":"Throughout the last decade, the so-called replication crisis has stimulated many researchers to conduct large-scale replication projects. With data from four of these projects, we computed probabilistic forecasts of the replication outcomes, which we then evaluated regarding discrimination, calibration and sharpness. A novel model, which can take into account both inflation and heterogeneity of effects, was used and predicted the effect estimate of the replication study with good performance in two of the four data sets. In the other two data sets, predictive performance was still substantially improved compared to the naive model which does not consider inflation and heterogeneity of effects. The results suggest that many of the estimates from the original studies were inflated, possibly caused by publication bias or questionable research practices, and also that some degree of heterogeneity between original and replication effects should be expected. Moreover, the results indicate that the use of statistical significance as the only criterion for replication success may be questionable, since from a predictive viewpoint, non-significant replication results are often compatible with significant results from the original study. The developed statistical methods as well as the data sets are available in the R package ReplicationSuccess.
","is_dataset_classified":null,"base_score":1.6094379124341003,"endowment":1.6094379124341003,"datacite_reuse_total":0,"file_count":0,"downloads":0,"views":0,"has_version_chain":false,"is_dataset":false,"is_oa":false,"pmid":"26657633","pmcid":null,"openalex_id":"https://openalex.org/W4205713785","authors":[],"funders":[{"funder_name":"Swiss National Science Foundation","grant_id":"189295","title":"Reverse-Bayes Design and Analysis of Replication Studies"}],"total_grants":1,"fwci":null,"citation_percentile":null,"influential_citations":0,"citation_trend":[{"year":2019,"count":1},{"year":2021,"count":1},{"year":2022,"count":1},{"year":2023,"count":1}],"oa_status":"gold","license":"cc-by","oa_locations":[{"url":"https://psyarxiv.com/fhwb7/download","host_type":""},{"url":"https://psyarxiv.com/fhwb7/download","host_type":""},{"url":"https://doi.org/10.31234/osf.io/fhwb7","host_type":""},{"url":"http://doi.org/10.31234/OSF.IO/FHWB7","host_type":"repository"},{"url":"https://osf.io/fhwb7","host_type":"repository"},{"url":"https://doi.org/10.1371/journal.pone.0231416","host_type":""},{"url":"https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0231416&type=printable","host_type":""},{"url":"https://www.zora.uzh.ch/id/eprint/195576/1/pawel-probabilistic-pone.0231416.pdf","host_type":""},{"url":"https://dx.doi.org/10.5167/uzh-195576","host_type":""},{"url":"https://pubmed.ncbi.nlm.nih.gov/32320420","host_type":""},{"url":"http://dx.doi.org/10.1371/journal.pone.0231416","host_type":""},{"url":"https://doaj.org/article/b2fc1f4e8913469f888b74d49f3dc5c8","host_type":""},{"url":"https://dx.doi.org/10.1371/journal.pone.0231416","host_type":""},{"url":"https://sonar.ch/global/documents/91327","host_type":""},{"url":"https://doi.org/10.5167/uzh-195576","host_type":""},{"url":"https://www.zora.uzh.ch/id/eprint/195576/","host_type":""},{"url":"https://psyarxiv.com/fhwb7","host_type":""}],"fields_of_study":["Scientific Computing and Data Management","Statistical Methods in Clinical Trials","Evolution and Genetic Dynamics","01 natural sciences","0101 mathematics"],"mesh_terms":[],"keywords":["Replication (statistics)","Inflation (cosmology)","Computer science","Probabilistic logic","Econometrics","Statistical model","Statistics","Artificial intelligence","Economics","Mathematics","Science","PsyArXiv|Meta-science","Social Sciences","610 Medicine & health","Genetics and Molecular Biology","1100 General Agricultural and Biological Sciences","1300 General Biochemistry, Genetics and Molecular Biology","Meta-science","610 Medicine & health","Probability","bepress|Life Sciences|Research Methods in Life Sciences","1000 Multidisciplinary","11559 Center for Reproducible Science","Research","Q","R","10060 Epidemiology, Biostatistics and Prevention Institute (EBPI)","General Medicine","Models, Theoretical","General Biochemistry","Medicine","General Agricultural and Biological Sciences","Software","Research Article"],"sdg_mappings":[{"sdg_number":0,"sdg_label":"Reduced inequalities"},{"sdg_number":0,"sdg_label":"Peace, Justice and strong institutions"}],"linked_datasets":[],"clinical_trials":[],"software_tools":[],"database_accessions":[],"source":"live","citation_network_status":"fetched"},"created_at":"2026-07-25T07:13:27.101349Z","pmid":null,"pmcid":null,"fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":null,"fair_a":null,"fair_i":null,"fair_r":null,"fair_zscore":null,"fair_rationale":null,"fair_model":null,"fair_agent_version":null,"fair_fulltext_source":null,"fair_has_llm":null,"fair_computed_at":null,"clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}