Satoki Ishikawa

『確率的機械学習』の発売日，IBISが始まる二日前なのは合わせたように見える👀

My favorite pianist Erik Lu got the first prize at the Chopin Piano Competition😀

IBIS，発表は間に合わなかったのですが，参加はすることにしました😀懇親会はもちろん間に合わなかったのですが，裏懇親会に申し込みました．よろしくお願いします🙇

はい😇jaxをtorchのように使いたいということでは，torchaxに期待しているのですが🤔

1つのライブラリ（torch, trl）に慣れてしまうと，例え他のライブラリ（jax, verl）の方が適していたとしても，移行する際の心理的な障壁がとても高い．プログラミング言語間の翻訳に特化したLLMが一番欲しい…

YouTube video by OpenAI Shipping with Codex

私も現状ではプロダクトを意識するレベルでは難しそうと思いつつ，Open AIの人たちが動画で1時間あたり約4000行のPRをテスト駆動開発（仮）で作っていると言っているのを見ると，設計書，テスト，検収条件などさえ正確に設計すれば，コードの細かい要件は考えずに，とにかくコードを書き続けるといったことも，そのうちにできるようになるのかなとも感じ，戦々恐々としています😱
www.youtube.com/watch?v=Gr41...

The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton Recent efforts to accelerate LLM pretraining have focused on computationally-efficient approximations that exploit second-order structure. This raises a key question for large-scale training: how much...

Practical upper-bound is an interesting concepts. What kind of practical upper bound would be interesting other than this?
arxiv.org/abs/2510.09378

Muon，論文にする場合は，早めに書かないと，また誰かと被る可能性がある一方で，何かあとひと押しのオリジナリティが出せない…

Jeremy Cohen on X: "This nice, thorough paper on LLM pretraining shows that quantization error rises sharply when the learning rate is decayed. But, why would that be? The answer is likely related to curvature dynamics. https://t.co/cdkt3DU1iw" / X This nice, thorough paper on LLM pretraining shows that quantization error rises sharply when the learning rate is decayed. But, why would that be? The answer is likely related to curvature dynamics. https://t.co/cdkt3DU1iw

While the focus for generalization and implicit bias has been on robustness to sample-wise noise, the rise of large-scale models suggests that robustness to parameter-wise noise (e.g., from quantization) might be now just as important?

x.com/deepcohen/st...

So many papers… it’s a bit overwhelming. Wish there were a field with fewer of them...

I’ve been challenging myself to read a lot of NeurIPS 2025 papers, but maybe I should switch soon to reading ICLR 2025 submissions instead.

This might be one of the advantages of methods that skip curvature EMA (like Muon) or use the function gradient (like NGD).

This paper is really interesting.
NGD builds curvature from the function gradient df/dw, while optimizers like Adam and Shampoo use the loss gradient dL/dw.
I’ve always wondered which is better, since using the loss gradient with EMA might cause loss spikes later in training.

This paper studies why Adam occasionally causes loss spikes, which is attributed to the edge of stability phenomenon. As seen from the figure, once hitting EOS (see b) a loss spike is triggered. An interesting experimental report!

arxiv.org/abs/2506.04805

I'm looking at ICLR submissions and I've noticed a significant number of papers related to Muon.

「スパース」という単語が関連する分野横断型のワークショップやると，お互いでどういう議論になるか興味がある．．．

学習ダイナミクスや暗黙的バイアスの観点から嬉しいスパース構造と，GPUを用いた行列積にとって嬉しいスパース構造と，脳が持っているスパース構造，どのくらいオーバーラップがあるのですかね🤔GPUを用いた行列積にとって嬉しいスパース行列の構造は複数パターン知られていますが，その学習理論や神経科学との接続はあまり聞かず，ただHPCの人も他分野に興味があるので，関連しそうな文献に引用は飛ばしつつも，あと一歩で行き詰まっている印象？

YouTube video by Chopin Institute KEVIN CHEN – first round (19th Chopin Competition, Warsaw)

And he is the genious.
www.youtube.com/watch?v=iZAp...

YouTube video by Chopin Institute SHIORI KUWAHARA – first round (19th Chopin Competition, Warsaw)

Everyone is giving a wonderful performance, but among the Japanese pianists, I'm particularly fond of Ushida-kun and Kuwahara-san.

www.youtube.com/watch?v=SPS4...
www.youtube.com/watch?v=DaY6...

YouTube video by Chopin Institute ERIC LU – first round (19th Chopin Competition, Warsaw)

I’m watching Chopin Competition.
I’m so surprised that he is using an office chair while I’m deeply impressed by his performance.

m.youtube.com/watch?v=fDsg...

二つの全く違うところでやっていた違うテーマ両方とオーバーラップが大きい研究が出てくると，精神的ダメージが大きいのですが，テーマ設定が安易すぎたのかもしれない

いいテーマがないか放浪する旅をします🧳

ここ2~3ヶ月考えていた２つのテーマに極めて近い論文が，昨日同時に２本出てしまい，研究テーマが消滅してしまった．．．早めに切り替えて，いい研究テーマをゼロから考え直さないといけないですね．．．

PINN，全然知らなかったのですが，最適化&Scalingの対象として面白そう．

Not all scaling laws are nice power laws. This month’s blog post: Zipf’s law in next-token prediction and why Adam (ok, sign descent) scales better to large vocab sizes than gradient descent: francisbach.com/scaling-laws...

I've made some small updates to the 'awesome list' for second-order optimization I made two years ago. It looks like Muon related works and the applications to PINNs have really taken off in the last couple of years.
github.com/riverstone49...

Fluid dynamics might serve as an interesting new benchmark for second-order optimization

Discovery of Unstable Singularities Whether singularities can form in fluids remains a foundational unanswered question in mathematics. This phenomenon occurs when solutions to governing equations, such as the 3D Euler equations, develo...

I don’t know anything about fluid dynamics, but I came across a paper that seemed to say that second-order optimization is key when using the power of neural networks to solve the Navier–Stokes equations. If so, there’s something romantic about that.
arxiv.org/abs/2509.14185

This is not OK.

I don't submit often to NeurIPS, but I reviewed papers for this conference almost every year. As a reviewer, why would I spend time trying to give a fair opinion on papers if it's what happens in the end???

ありがとうございます！