{"doi":"10.31234/osf.io/9y3mp","title":"Null-hacking, a lurking problem","abstract":"<p>Pre-registration of analysis plans involves making data-analysis decisions before the data is run in order to prevent flexibly re-running it until a specific result appears (p-hacking). Just because a model and result is pre-registered, however, does not make it reflective of underlying reality. The complement to p-hacking, null-hacking, is the use of the same questionable research practices to re-analyze open data to return a null finding. We provide a vocabulary for null-hacking and introduce the threat it poses. Null-hacking forces consideration of model fit to compare pre-registered and ‘alternative’ models. The reason null-hacking cannot be ignored is a null-hacked model can easily provide better fit to the data than a pre-registered one. Model fit, however, is a precarious problem, focusing just on model fit by only selecting a ‘best fitting model’ eliminates pre-registration, while giving default preference to pre-registered results ignores how well our models can represent the data. We provide a beginning solution aimed at retaining the advantage and justifications of pre-registration, while including model fit, and providing protection against null-hacking. We call this Fully-Informed Model Pre-registration and it involves strict supervised machine learning to maximize local model fit within heavily pre-specified decisions. This solution maximizes local model fit, eliminating the only justifications null-hacked results have. It is not yet a complete solution but merely the groundwork for why other approaches may be insufficient and what a future solution may look like.</p>","journal":null,"year":2018,"id":1062,"datarank":0.8724996711459398,"base_score":2.639057329615259,"endowment":2.639057329615259,"self_citation_contribution":0.3958585994422889,"citation_network_contribution":0.4766410717036509,"self_endowment_contribution":0.3958585994422889,"citer_contribution":0.4766410717036509,"corpus_percentile":null,"corpus_rank":null,"citation_count":13,"citer_count":11,"citers_with_citation_signal":9,"citers_with_endowment":9,"datacite_reuse_total":0,"is_dataset":false,"is_dataset_confidence":0.0423,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2018-06-21","fair_score":null,"fair_percentile":null,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":14635,"name":"John Protzko","orcid":"0000-0001-5710-8635","position":0,"is_corresponding":true}],"reference_count":55,"raw_metadata":{"citation_network_status":"fetched"},"created_at":"2026-03-01T18:20:47.508186Z","pmid":null,"pmcid":null,"fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":null,"fair_a":null,"fair_i":null,"fair_r":null,"fair_zscore":null,"fair_rationale":null,"fair_model":null,"fair_agent_version":null,"fair_fulltext_source":null,"fair_has_llm":null,"fair_computed_at":null,"clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}