Miklos Bognar

Much thanks to the amazing team: @martonaronvarga.bsky.social , @donvanraven.bsky.social , @kekecszoltan.bsky.social, @jimgrange.bsky.social, @balazsaczel.bsky.social & Máté Gyurkovics

22.09.2025 08:16 — 👍 0    🔁 0    💬 0    📌 0

We think that research fields where notable "ground truth" effects are investigated (such as the CSE), a similar systematic exploration of the analytical space is necessary to inform the field's community about common arbitrary decision combinations that can lead to higher false findings.

20.09.2025 07:08 — 👍 1    🔁 0    💬 1    📌 0

Based on these results we think that the risks of multiple testing (even with common corrections) are higher than expected, thus sticking to a preregistered analytical protocol is immensely recommended.

19.09.2025 13:06 — 👍 0    🔁 0    💬 1    📌 0

in repeated-measures ANOVAs, FPRs were not affected by outlier filtering methods; thus, when severe outlier filtering is justified, repeated-measures ANOVA is a recommended choice for hypothesis testing.

19.09.2025 13:06 — 👍 0    🔁 0    💬 1    📌 0

In linear models, type I error rates also increase proportionally to the severity of outlier filters. This inflation of FPR poses a significant risk of false findings; therefore, we do not recommend to use linear mixed models along with severe outlier exclusion techniques, especially on skewed data.

19.09.2025 13:06 — 👍 0    🔁 0    💬 1    📌 0

Model TPRs on large effect size datasets on different participant numbers, with the 3SD outlier filtering method. True positive rate is indicated on the y-axis, while false positive rate is indicated on the x-axis. Hypothesis testing models are shown with different colors, and numbers on the plot indicate different sample sizes. An assumed maximum FPR of.025 is indicated with a dashed vertical line

Results showed that certain analytical choice combinations (outlier filtering; data transformation; hypothesis testing method) led to highly inflated false positive rates (type I error rates). Decision pathways where linear mixed-effect models were used were especially impacted.

19.09.2025 13:06 — 👍 0    🔁 0    💬 1    📌 0

Multiverse simulation to explore the impact of analytical choices on type I and type II errors in a reaction time study - Behavior Research Methods Researcher degrees of freedom in data analysis present significant challenges in social sciences, where different analytical decisions can lead to varying conclusions. In this work, we propose an exam...

I am happy to announce the publication of our new work on the impact of arbitrary analytical choices on type I and type II error rates. We simulated reaction time data in a conflict task and analyzed the notable CSE effect in a multiverse manner. Worrying results:
link.springer.com/article/10.3...

19.09.2025 13:06 — 👍 14    🔁 6    💬 1    📌 0