Giacomo Bignardi

I've added a new example to our paper's repo, demonstrating how our reliability method replicates Cronbach's alpha for a simple model, but also how our method can account for: (i) binary data, (ii) varying numbers of items per pps, & (iii) improvement over trials www.bignardi.co.uk/8_bayes_reli...

Thanks Edwin! I only learned from the best 😜

Great! Sorry just saw this before my replies :)

I think both methods should converge to the same answer as the number of draws -> ∞. I think your approach is 1-V_w/V whereas empirical reliability is defined as v_a/(v_a+v_w) - but it should be the same as draws -> ∞. Pic from onlinelibrary.wiley.com/doi/10.1002/...

Difference between empirical and marginal reliability of an IRT model I am using the mirt library in R to fit an instrument (binary responses) comprising two dimensions. In the mirt documentation are mentioned two types of reliability. I would like to ask what is the

Hi Ruben - i had the same thought - and it wasn't easy finding the reference in an old IRT manual! In your post, do you divide the variance of subjects' posterior means by the total variance in MCMC draws across subjects? The best description of ER is in here by Phil Chalmers tinyurl.com/4f3vv5e8

...

Any (ideally helpful) comments/feedback/critique welcome! Email at: bignardig@outlook.com

Alongside the preprint osf.io/h54k8 I've uploaded an example of how to calculate RMU with go/no-go task data, comparing RMU with split-half and test-retest reliability estimates www.bignardi.co.uk/8_bayes_reli... - more software support to come if there's demand.

In the paper, we also demonstrate that our method yields a similar point estimate to using the ratio of subjects' posterior mean variance (PMV) divided by PMV + average posterior variance. However, our approach also provides credible intervals, which are essential when sample sizes are small.

We also compared RMU to coefficient alpha and H (Study 1) and split-half reliability (Study 2). RMU generally had lower bias and error, as well as better coverage. However, the main benefit of RMU is that it can be applied across a wide range of Bayesian measurement models - unlike alpha.

We ran 3 simulation studies to evaluate RMU's bias, accuracy and coverage (% of times the 95% credible intervals included the right answer) across linear factor, signal detection theory and reinforcement learning models. Results were good-to-great across studies and simulation conditions.

Our method involves taking two random draws (without replacement) from each subject's posterior and computes the correlation between draws. We repeat this process many times and use the distribution of correlations to calculate credible intervals or posterior probabilities (e.g., reliability > .70).

The intuition behind our method is that the amount of overlap between posteriors is important: in the example above, if the posteriors became significantly broader, we would be much more uncertain in each estimate and struggle to distinguish between participants, reducing reliability.

Our reliability method can be applied to most fitted Bayesian measurement models. In the example below, I've used a Bayesian signal detection model to estimate posterior distributions for each subject's performance on a go/no-go test. Each curve shows our certainty in each subject's estimate.

Individual differences in computational psychiatry: A review of current challenges - PubMed Bringing precision to the understanding and treatment of mental disorders requires instruments for studying clinically relevant individual differences. One promising approach is the development of com...

While estimating reliability with questionnaires is straightforward, there are few methods when deriving individual differences from computational models. Thus, reliability is often poor for widely used computational measures when evaluated with retest data (e.g., pubmed.ncbi.nlm.nih.gov/36940888/ )

New preprint with @rogierk.bsky.social @paulbuerkner.com - we introduce "relative measurement uncertainty" - a reliability estimation method that's applicable across a broad class of Bayesian measurement models (e.g., generative-, computational- and item response theory-models osf.io/h54k8

...

Inspecting the regression coefficients, we found that going to bed hungry was the strongest and most consistent predictor out of all the food insecurity questions.❗We should note that the survey took place during school term, and food insecurity is likely even worse during the school holidays.

📈Comparing model predictions for low food insecurity to medium or high food insecurity, we found moderate (.3-.5 standard deviations) changes in depression and adolescent wellbeing scores, and smaller changes (.2-.3 standard deviations) in anxiety, loneliness and positive thoughts.

Using #marginaleffects, we compared model predictions for each outcome variable across three levels of food insecurity: "low", "medium", and "high". Each level represents an "average" participant who either responded "never", "sometimes", or "often" to each food insecurity question.

Next, we used ordinal regression models to estimate the association between the three food insecurity questions and six mental health outcomes, controlling for demographics. We modelled the effect of each food insecurity question separately (and modelled "sometimes" and "often" separately).

Table showing the prevalence of food insecurity responses across three questions, split into "never", "sometimes" and "often" responses.

We found 4-5% of CYP sometimes experience food insecurity, and 1-2% often experience food insecurity, across our three questions (F1: food bank usage, F2: food affordability, F3: going to bed hungry). Post-stratification weighting using school census data did not massively alter these estimates.

JCPP Advances | ACAMH Child Development Journal | Wiley Online Library BackgroundWe aimed to assess the prevalence of food insecurity reported by children and young people in four areas of England in 2023 and examine its association with mental health and wellbeing. Met...

📣New study 📣 We examined the prevalence of food insecurity and its association with mental health and wellbeing, using the 2023 @oxwell.bsky.social survey of 38,430 children and young people aged 8–19 in England with @sjblakemore.bsky.social @minafazeloxford.bsky.social doi.org/10.1002/jcv2...

New paper! Excellent work by @bignardi.bsky.social and @minafazeloxford.bsky.social, who leads the Oxwell study

The prevalence of food insecurity and its relationship with wellbeing in a large, cross-sectional study of children and young people in England

Out soon. Preprint:
osf.io/preprints/ps...

Delighted to be hiring a new Clinical Psychology Lecturer to come and join our vibrant environment at the SGDP Centre @kingsioppn.bsky.social @kingscollegelondon.bsky.social please share!

my.corehr.com/pls/coreport...

You could use a hot deck method like predictive mean matching which uses other data points on the variable to impute - so you shouldn't get out of range values?

I've just tried to stick to terminology from marginaleffects.com and explain the specifics in the methods. It was a bit of a pain when it comes to peer review as our reviewers were not familiar with marginal effects... still ploughing on though with using it in more upcoming papers though 😀😬

Your pics just look black to me! Lets see if this works...

@drandreahoward.bsky.social, hold my beer..

Latent class growth models are worse than useless, and we've known this for more than 20 years.

(See Bauer 2007)

I wonder to what extent this reflects changing terminology - "wellbeing" research is skyrocketing. The content of "wellbeing" and "happiness" scales seems to overlap quite a bit too.