Michael Poli

[6] It's now undeniable that, with a little bit of creativity, improving scaling is not only approachable but also particularly rewarding. And while I'm obviously excited by convolution-attention hyena hybrids due to their balance of efficiency and quality across domains, there's a lot more to do!

14.11.2024 20:33 — 👍 0    🔁 0    💬 0    📌 0

[5] We have seen time and again that various classes of computational units outperform others in different modalities, on different tasks, in different regimes. We've seen this in scaling laws, on synthetics, on inference.

14.11.2024 20:32 — 👍 0    🔁 0    💬 1    📌 0

[4] There has been a flurry of work over the last couple of years (from the great people at HazyResearch and elsewhere) on developing bespoke model designs as "proofs of existence" to challenge the Transformer orthodoxy, at a time when model design was considered "partially solved."

14.11.2024 20:32 — 👍 0    🔁 0    💬 1    📌 0

[3] We continue to push the scale of what's possible with "beyond Transformer" models applied to biology, in what could be among the most computationally intensive fully open (weights, data, pretraining infrastructure) sets of pretrained models across AI as a whole.

14.11.2024 20:31 — 👍 0    🔁 0    💬 1    📌 0

[2] A lot has happened since the first release of Evo. We have made public the original pretraining dataset (OpenGenome)—links below—and will soon release the entire pretraining infrastructure and model code.

14.11.2024 20:31 — 👍 0    🔁 0    💬 1    📌 0

An absolute privilege to see our work on Evo 🧬 highlighted on the cover of the latest issue of Science. Thank you to all the friends and collaborators at Stanford and the Arc Institute.

14.11.2024 20:30 — 👍 2    🔁 0    💬 2    📌 0