the people angry about the huggingface scraping are not long term thinkers. we are talking about people who have not even begun to think about what the second order consequences of highly centralized information access look like

(context)

Elon the type of guy to actually consider building the Absolutely Safe Capsule from Mother 3

i always come fast

"free culture for me but not for thee"

*the lack of learned optimization

there is a ton of information in the past changes of gradients that we could be doing more with than running averages!

i also think learned optimization may end up being far more of a bottleneck in the long term compared to the architectural structure of neural networks in terms of sample efficient learning

arxiv.org/abs/1606.04474

Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process Recent advances in language models have demonstrated their capability to solve mathematical reasoning problems, achieving near-perfect accuracy on grade-school level math benchmarks like GSM8K. In thi...

arxiv.org/abs/2407.20311

clearly we can observe, that the deeper the network is, the better the heuristics that form in the network when it comes to generalizing to "like data".

so blanketly describing the solutions that dnns make as "poorly generalizable" is a little bizarre to me tbh

yeah i respect him a lot more than Gary Marcus in this regard. i just think there's a lack of humility involved when we abstract things to reductionist observations on the level of, "its just curve fitting bro"

i mean, how can you coherently assess the formation of approximated functions that don't actually exist in the data as a form of "memorization" if the internal heuristics of the network look nothing like the data but are formed by an attempt to match it?

yeah i think more broadly this is a useful thing to think about for the record

there are some people who are extremely dead set on their current interpretation of how dnns "actually" work (Gary Marcus, François Chollet) and refuse to look through any other lens

pretty unfortunate

a what now?

does attending to the attention scores of the "happy" token imbue a consistent expression of this semantically? does it only imbue it for a small subspace of attention comparisons?

assessing this seriously forces you to unpack a black box of unfalsifiable metaphysics questions

my objection to this is not on the basis of humans being "special" but on the basis of Transformers being state simulators

whenever Claude says it is "happy", if most of the distribution it did a PRNG diceroll over is an equal spread of emotions, how can we say that there was intent to express it?

the world is a ghetto with big guns and picket signs,

but it can do whatever it want, whenever it want, i don't mind

the dynamics of public posts on a decentralized social media network are nothing like the dynamics of physically owned private property

The Basilisk Cometh

can someone please eloquently explain to me how copying the posts from a database into another database is violence, or otherwise allows for exorbitant harms?

the vague insinuations of this being a threat to people's safety is especially ???
wtf is this leap in logic?

getting mad at the thing relative to what harm it actually causes instead of on the principle of it being for AI would require critical thinking instead of acting like a reactionary and rolling with my gut instinct though.

this is the logical consequence of the design of the website. it was always meant to be open. the firehose in particular is completely free

you guys do realize that all the leading companies were gonna do this privately regardless whether or not you asked politely, right?
and the only difference is, this is public and researchers/hobbyists can tinker with it more easily?

ever since i read the bluesky api was completely open, i knew someone was gonna do this and people would respond with the most artificial faux outrage possible

"publicly available data was used by people when i posted it online. this is an ethical crime,"

a cartoon character wearing a purple hat and gloves is jumping over a checkered floor . ALT: a cartoon character wearing a purple hat and gloves is jumping over a checkered floor .

deep learning is hitting a wall but what if if hit on ME instead?