Luke Guerdan

Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks Data scientists often formulate predictive modeling tasks involving fuzzy, hard-to-define concepts, such as the "authenticity" of student writing or the "healthcare need" of a patient. Yet the process...

📄 arxiv.org/abs/2507.02819

This work was in collaboration with the amazing team @devsaxena.bsky.social (co-first author), @schancellor.bsky.social, @zstevenwu.bsky.social , and @kenholstein.bsky.social

Thank you for making my first adventure into qualitative research a delightful experience :)

Our paper offers design implications to support this, such as:

- Protocols to help data scientists identify minimum standards for validity and other criteria, tailored to their specific application context
- Tools designed to help data scientists identify and apply strategies more effectively

The challenge for HCI, CSCW, and ML is not to *replace* these bricolage practices with rigid top-down planning, but to develop scaffolding that enhances the rigor of bricolage while preserving creativity and adaptability

Yet from urban planning to software engineering, history is rife with examples where rigid top-down interventions have failed while bottom-up alternatives designed to better scaffold *existing* practices succeeded

What do these findings mean for how we improve target variable construction going forward? We might be tempted to more stringently enforce a rigid "top-down planning approach" to measurement, in which data scientists more carefully define construct → design operationalization → collect data

How do data scientists evaluate validity? They treat their target variable definition as a tangible object to be scrutinized. They "poke holes" in their definition then "patch" them. They apply a variety of "spot checks" to reconcile their theoretical understanding of a concept with observed labels

Data scientists navigate this balancing act by adaptively applying (re)formulation strategies

For example, they use "swapping" to change target variables when the first has unanticipated challenges, or "composing" to capture complementary dimensions of a concept being captured in a target variable

An illustration of the target variable construction process presented in our findings. During target variable construction, data scientists specify an initial prediction task based on their available data, then iteratively refine their prediction task by applying (re)formulation strategies. Data scientists proceed with their final prediction task if it satisfies all criteria, or discontinue their project if strategies are exhausted.

While engaging in bricolage, data scientists balance the validity of their target variable with other criteria, such as:
💡 Simplicity
⚙️ Resource requirements
🎯 Predictive performance
🌎 Portability

We find that target variable construction is a *bricolage practice*, in which data scientists creatively "make do" with the limited resources at hand

To explore this tension, we interviewed 15 data scientists from education and healthcare sectors to understand their practices, challenges, and perceived opportunities for target variable construction in predictive modeling

Traditional measurement theory assumes a top-down workflow, where data is collected to fit a study's goals (define construct → design operationalization → collect data)

In contrast, data scientists are often forced to reconcile their measurement goals with *existing* data

A subtle aspect of predictive modeling is target variable construction: the process of translating a latent, unobservable concept like "healthcare need" into a prediction target

But how does target variable construction unfold in practice, and how can we better support it going forward? #CSCW2025 🧵

✨I’m on the academic job market ✨

I’m a PhD candidate at @hcii.cmu.edu studying tech, labor, and resistance 👩🏻‍💻💪🏽💥

I research how workers and communities contest harmful sociotechnical systems and shape alternative futures through everyday resistance and collective action

More info: cella.io

Screenshot of the CSCW 2025 paper "The Future of Tech Labor: How Workers are Organizing and Transforming the Computing Industry" CELLA M. SUM, Carnegie Mellon University, USA ANNA KONVICKA, Princeton University, USA MONA WANG, Princeton University, USA SARAH E. FOX, Carnegie Mellon University, USA Abstract: The tech industry’s shifting landscape and the growing precarity of its labor force have spurred unionization efforts among tech workers. These workers turn to collective action to improve their working conditions and to protest unethical practices within their workplaces. To better understand this movement, we interviewed 44 U.S.-based tech worker-organizers to examine their motivations, strategies, challenges, and future visions for labor organizing. These workers included engineers, product managers, customer support specialists, QA analysts, logistics workers, gig workers, and union staff organizers. Our findings reveal that, contrary to popular narratives of prestige and privilege within the tech industry, tech workers face fragmented and unstable work environments which contribute to their disempowerment and hinder their organizing efforts. Despite these difficulties, organizers are laying the groundwork for a more resilient tech worker movement through community building and expanding political consciousness. By situating these dynamics within broader structural and ideological forces, we identify ways for the CSCW community to build solidarity with tech workers who are materially transforming our field through their organizing efforts.

What can #CSCW learn from tech workers who have been involved in collective action and unionization about how to make transformative change within our field?

My new #CSCW2025 paper with Mona Wang, Anna Konvicka, and Sarah Fox seeks to answer this question.

Pre-print: arxiv.org/pdf/2508.12579

You are eligible to participate if you have experience designing evaluations that use both (1) an LLM-as-a-judge and (2) a rubric to rate GenAI outputs. We welcome participants from all professional roles. Participants must be 18+ and be located in the U.S.

Have you built a generative AI evaluation that uses an LLM-as-a-judge and a rubric to rate model outputs?

Sign up for a 45-minute Zoom session to provide feedback on a new tool for building trustworthy evals.

Learn more at tinyurl.com/llm-as-a-judge - receive $35 for participating in a session!