Toni Kukurin

agree on the latter point, cf bsky.app/profile/tkuk...

on the former - it's costly if one assumes grok invocation per page load. but IMO scaling a grok-twitter-recsys is amenable to good system dimensioning (e.g. model cascade + periodic batching)

from that perspective, doesn't seem preposterous 🤷

how would satya sell enterprise AI without text extraction

our entire economy depends on document scanning for training data.

iirc @neuroai.bsky.social make a nice remark at some point during the learning salon. paraphrasing from memory, "is the real AGI DeepMind's accumulated bias + methods or are we _truly_ building a general learner"

"long-context" and "prompt eng" are flip-sides of the same underlying concept. _someone_ has to compress the representation, either in language- or high-d-vector space.

IMO the domain model is intelligence as part user (akin to "phds doing lin reg" in finance), part interface (RLHF), part model

interesting but I guess not a relevant attack vector (assuming it exclusively influences user-local results).

becomes relevant in some intricate global-recommender-system situation but google already has years of dealing with this behind them.

how do you here define _frontier work_? most work I do at least seems a combination of monotonous incantations and real insight. insight usually being ~ "this will affect outcomes I care about N steps down the road". && frequently this is preempted by me just-realizing which outcomes I care about :)

but also sounds like a cool research project. rev-eng how much google3 code in gemini pre-training based on relative quality of sth like gemini vs gpt4 or claude.

(eg via diff-in-diffs perf on coding tasks nodejs vs swift)

(I assume gem was predominantly google3-trained based on its coding quirks)

Open-Endedness is Essential for Artificial Superhuman Intelligence In recent years there has been a tremendous surge in the general capabilities of AI systems, mainly fuelled by training foundation models on internetscale data. Nevertheless, the creation of openended...

novelty = f(audience, goal)?

there's a nice exposition of this topic in @rockt.ai and others' position paper arxiv.org/abs/2406.042...

which, to prove a meta-point, delivers a rather "obvious" message with exposition clarity deemed novel enough for ICML poster acceptance :)

"dynamic" as discussed here sounds to me more akin to what people would refer to as "reasoning" fine-tuning nowadays?

RAG implies external memory; reasoning post-trained model generates more artefacts (/tokens/"writes") as a result of computation (which is also where "dynamic" makes a difference)

FineWeb: decanting the web for the finest text data at scale - a Hugging Face Space by HuggingFaceFW This application helps create large, high-quality datasets for training large language models by processing and filtering web data. Users can obtain datasets with optimized deduplication and filter...

not sure about a single unifying tutorial textbook; however closest might be something in the huggingface/anthropic/... blogpost genre... eg. what comes to mind in data filtering/collection space
huggingface.co/spaces/Huggi...

Humans demonstrating OOD failure, AI demonstrating IID success :)

bsky.app/profile/mela...

YouTube video by Krzysztof A. Janczak A message from Ira Glass that every Artist should hear...

"its normal and the most important thing you can do is do a lot of work" :)

it would be intriguing to see the progress if you're willing to share at some point!

related and belated followup: opinion/rant on similar topic

> Every app has a different design [optimized based on your activity ...] each app trying to make you do different things in uniquely annoying ways [...] low-quality clickbait

anyhow we all know how it goes

YouTube video by Case Western Reserve University Kurt Vonnegut Lecture

I mean, you've described a lot of content. "low entropy" does some moderately-heavy lifting, but in essence www.youtube.com/@kurzgesagt videos are also this way.

all of art ultimately exists on a particular grounded low entropy axis (nice example by Kurt Vonnegut youtu.be/4_RUgnC1lm8?...)

agree re/feeds

what you'd normally do for a closed-platform is (1) scrape posts, (2) write-your-own ranking/clf. algo to get better content (tags are useful but easily co-opted, the platform's recsys is necessarily deficient due to partial obs).

_native_ support for scaling (2) I find exciting

💯, people complain to overuse of mathematical formalisms in papers but a less expressive language enforces precision.

(in which sense, I guess a similar analogy exists: internal monologue vs. writing)

How was it, "if I had more time I would've written a shorter letter"... Cursor has all the time in the world, now to incentivize compression. Twitter v0 had the right idea :)

GitHub - twitter/the-algorithm: Source code for Twitter's Recommendation Algorithm Source code for Twitter's Recommendation Algorithm - twitter/the-algorithm

despite all those heroic efforts to open source "the algorithm"?! shocker

How many turns into the conversation? Any way to share the full trace?

composition (in standard unixy sense), decomposition (use part of functionality and pipe into another program), misappropriation (eg use excel as coloring tool), environment-driven personalization (e.g. use OS-level accessibility tools)

+1, I conceptualize as generalized state of flow beyond visceral fleeting moments. prob why env design (school, "WFH v onsite", etc) matters & likely down to energy expense: helper.ipam.ucla.edu/publications...

> Control energy for state transitions decreases over the course of repeated task trials

are you curious about oai index/retrieval quality or aiming at something else?

Could you share a few? I feel like Goodhart is well interpreted in CS/ML, curious for specific counterexamples.

the data transform + API integration is great, but how do I tell it about function composition :'(

also great model of why modern LLMs work.

user intent is specified in the query.

informed re-representation (his example: "cat chased by dog" vs "dog chases cat") massively benefits info transmission even when no new information is generated.

@andymatuschak.org's writing is relevant (both on learn-to-learn e.g. [1] and personal xp, e.g. timely [2]). I think immediate-feedback learning style should become default w/LMs. quick feedback loops is how you win any system :)

[1]: andymatuschak.org/prompts/
[2]: bsky.app/profile/andy...

what were the assignments about? encouraging review via daily journal habit (e.g. essay "what we learned today") since early ed seems immensely useful.

Does it fully matter? Student learns to use AI tools. Could be two way learning. Teacher is around mainly to enforce meta cognition and reflection.

yes! agree it's a different psych manifestation.

but AFAIK "decision fatigue" usually implies deteriorating decision quality.

so counterfactual is ~ "did my problem-solving deteriorate less after 8h study relative to my decision making ability after 8h work".

which... not sure if true.