Item Response Warehouse

Abstract of the paper.

Title page of the paper.

The Item Response Warehouse is a new data resource for psychometricians interested in developing methods using bigger and more diverse sets of instruments: itemresponsewarehouse.org

New paper out now at BRM: doi.org/10.3758/s134...

An introduction to the Item Response Warehouse (IRW): A resource for enhancing data usage in psychometrics - Behavior Research Methods The Item Response Warehouse (IRW) is a collection and standardization of a large volume of item response datasets in a free and open-source platform for researchers. We describe key elements of the da...

Publication describing the IRW now out! link.springer.com/article/10.3...
Over 1000 datasets in there with tools for pulling directly from R!

Lots of item-level heterogeneity in treatment sensitivity across a wide-range of RCTs (consistent with the earlier findings from www.tandfonline.com/doi/abs/10.1... that focused on education-based RCTs).

🔥Coming July 2: 'Metrics and Models'!🔥

A new seminar series on advanced modelling w/ impact across health/society.

🌍 Open to *ALL*!
🕑 Weds 2pm UK alternating weeks (online)
🔗 metrics-and-models.github.io
📬 metrics_and_models-subscribe@maillist.ox.ac.uk

Brilliant speakers. *Please share*!

Analysis of treatment effects on psychological networks using data from the IRW: "Our results show that causal effects on network strength are both common and uncorrelated with effects on network state." New work from @jbgilbert.bsky.social

psycnet.apa.org/doiLanding?d...

As the IRW approaches 1000 tables, there is an obvious need for better approaches when querying appropriate tables. We've added more metadata (including human annotation) and improved R-based functionality for doing just that! itemresponsewarehouse.org/metadata.html

Simulation studies are common in psychometrics. We are trying to make them more relevant to practice by increasing their connection to empirical data. Feedback welcome!
osf.io/preprints/ps...

New paper out yesterday in Behavior Research Methods (BRM): link.springer.com/epdf/10.3758...

@itemrespwarehouse.bsky.social

That leader being @sachaepskamp.bsky.social !

For those with psychometric interests: spread the word about this talk on network psychometrics from one of the leaders in that field this coming Wednesday. Details here: ssps.su.domains

See also additional description in section 4.4 of the updated IRW preprint. osf.io/preprints/ps...

We're working to make code available to make these kinds of analyses easy for others to do. For the item-level HTE work, see here: github.com/ben-domingue...

7/ Many of these data are available for reuse on the IRW! Next week we'll share code to help you start replicating these analyses!

6/ Understanding whether an empirical study is one versus the other may help us refine our understanding of what is really happening in the treatment.

5/ This paper shows that many treatments increase the response averages (ie a conventional treatment effect) while others serve to tighten connections between items (while not changing the overall level of item response).

4/ A separate paper (osf.io/preprints/ps...) pushes further to consider treatment effects from these studies from the perspective of network psychometrics.

3/ It goes further and shows that we may misunderstand uncertainty if we do not account for item-level variation and demonstrates identification challenges associated with the estimation of treatment-by-covariate interaction effects (a big deal given interest in heterogeneous treatment effects).

Heterogeneity of Item-Treatment Interactions Masks Complexity and Generalizability in Randomized Controlled Trials Researchers compare aggregate test data across treatment and control group in randomized controlled trials to determine intervention impact on student achievement. We show that item-level data and ...

2/ This study, along with previous work (www.tandfonline.com/doi/abs/10.1...), shows that item-level variation in treatment effects is a real problem in empirical data!

Estimating Heterogeneous Treatment Effects with Item-Level Outcome Data: Insights from Item Response Theory Analyses of heterogeneous treatment effects (HTE) are common in applied causal inference research. However, when outcomes are latent variables assessed via psychometric instruments such as educational...

1/ What might data-rich #psychometrics research look like? As a first example, consider @jbgilbert.bsky.social on heterogeneous treatment effects (HTE). Josh uses #irt to examine item-level HTE, with insights drawn from 75 datasets across 48 randomized trials (arxiv.org/abs/2405.00161).

See website for more details!
datapages.github.io/irw/

You can access these datasets straight from R or Python using API calls. Paperwork (citation & license) on the provenance of all inclusions available too.

Do you need educational and psychological item response data to do your research? The IRW has 600 item response datasets (more coming!), distributed in a standardized format and ready for analysis.