Kyle Lo

learning how to do something is a first-order use case for LMs, the development bottleneck has been collecting data covering a wide diversity of topics, until now ✌🏻

incredibly fun project led by our intern yapei chang

we mined the web for thousands of real-world “how to do X” step by step instructions and turned it into a dataset, synth data training procedure, eval suite, etc.

lol rip 😮‍💨

It’s like a score calculated against gold reference citations in generated lit review, so even humans don’t score high. i think the eval is saturated cuz so much subjectivity in what counts as appropriate citation. better phrasing is maybe that the citations are sensible up to some X

they’re separate poorly named systems lol 😂 Separate projects approaching same problem from different angles. Scholar QA approach from agentic system design, use whatever model. Ope Scholar approach from model-first, very light on system. The teams are working together to fuse ideas

our open model proving out specialized rag LMs over scientific literature has been published in nature ✌🏻

congrats to our lead @akariasai.bsky.social & team of students and Ai2 researchers/engineers

www.nature.com/articles/s41...

0 days since last mixup of eval results between "copa" (choice of plausible alternatives) & "coqa" (conversational QA) tasks 😐

The 5th Generation, Evaluation, and Metrics (GEM) Workshop will be at #ACL2026!

Call for papers is out. Topics include:
🐟 LMs as evaluators
🐠 Living benchmarks
🍣 Eval with humans
and more

New for 2026: Opinion & Statement Papers!

Full CFP: gem-workshop.com/call-for-pap...

mm yea i think that's always the case w productivity tools.

imo ability to adopt new tools is core part of the job. just like transition from plain text editors to IDEs, from sending files via FPT to using git for collab, from ad hoc Makefiles to package managers, etc. AI is just the latest thing

my concern is the growing pool of "unknown unknowns" as i interact less with code directly.

imo probably why i subconsciously have been leaning toward cursor over claude code or similar agents, even if the latter has a higher code-to-keystrokes ratio

i dont feel worse at this even if im not writing papers from-scratch as much as during early career

but coding feels different due to mismatch between what i express to the system (english) and what the system returns (code). i've already realized some gaps in libraries I used to know well.

whether my ability to review code will degrade as I offload increasingly larger workloads to AI

of course, this shift is present in other forms of generation, like paper writing, where my role has shifted to reviewing/editing (student's) drafts.

some thoughts about skill degradation w/ AI coding

im onboard w views that "english is the new programming language" & "software engineering", translating ambiguous goals to technical specs/execution, is still a skill.

im more concerned w shift from my role as a writer to a reviewer and

lucky to chat w sen. patty murray about olmo & importance of fully open AI

using opus to extract research topics from papers & it was giving me useless words like "training", "datasets", and "evaluation"

kept prompting it w examples of more informative topics and it ended up with "LLM training", "LLM datasets", and "LLM evaluation"

thx

yo endorse me for python skills

just realized ive had food on my face all day & nobody at office told me, thx ai2 frens 😫

u gotta shitpost more maria, ur content too informative 😆

i appreciate bsky has less AI product advertising; i do want to see more memes/shitposting/fun stuff and insights from industry/open source sphere, even if they dont have an attached paper

amaazinggg thxx 🙏🙏🙏

ive been clicking around in UI but i cant find it 😭 pls help

bsky wish list

i like the idea of different feeds but i actually want my subscription to select feeds to be taken as a preference signal ("more like this") that informs a "home/default" feed.

i really dislike the UX of having to tab through each subscribed feed, esp when there's also post overlap

some notion of 'views/impressions'? it kinda sucks to post and only see a couple of likes & no replies. if there's some intermediate signal that shows people at least read the post, that'd incentivize more imo

sports 🏈

nope just an admirer of room 003

just in case it wasn’t clear which room this is

some citation graphs data pipelines will create new "paper" nodes based on extracted bibstrings from PDFs

so in 2026 the papers we hallucinated in 2025 might end up being "real" papers on gscholar or sthn lol

ya ur rite! we'll update it ✌️

just had hechalou’s yin yang milk tea and i think i’ve transcended 🤤

Olmo 3.1 - a allenai Collection The latest members of the Olmo 3 family: another 3 weeks of RL for 32B Think, the 32B Instruct model, large post-training research datasets...

new olmo 3.1 artifacts: huggingface.co/collections/...
paper (arxiv soon): allenai.org/papers/olmo3
demo: playground.allenai.org

paper has:
🐟 more on our eval ideology
🦈 more baselines
🍣 more about RL Zero
etc

we picked final model (internally called moonlit surfer 🌛🏄) not just on bench scores but good vibes 🥰