Satoki Ishikawa

The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton Recent efforts to accelerate LLM pretraining have focused on computationally-efficient approximations that exploit second-order structure. This raises a key question for large-scale training: how much...

Practical upper-bound is an interesting concepts. What kind of practical upper bound would be interesting other than this?
arxiv.org/abs/2510.09378

Muon，論文にする場合は，早めに書かないと，また誰かと被る可能性がある一方で，何かあとひと押しのオリジナリティが出せない…

Jeremy Cohen on X: "This nice, thorough paper on LLM pretraining shows that quantization error rises sharply when the learning rate is decayed. But, why would that be? The answer is likely related to curvature dynamics. https://t.co/cdkt3DU1iw" / X This nice, thorough paper on LLM pretraining shows that quantization error rises sharply when the learning rate is decayed. But, why would that be? The answer is likely related to curvature dynamics. https://t.co/cdkt3DU1iw

While the focus for generalization and implicit bias has been on robustness to sample-wise noise, the rise of large-scale models suggests that robustness to parameter-wise noise (e.g., from quantization) might be now just as important?

x.com/deepcohen/st...

So many papers… it’s a bit overwhelming. Wish there were a field with fewer of them...

I’ve been challenging myself to read a lot of NeurIPS 2025 papers, but maybe I should switch soon to reading ICLR 2025 submissions instead.

This might be one of the advantages of methods that skip curvature EMA (like Muon) or use the function gradient (like NGD).

This paper is really interesting.
NGD builds curvature from the function gradient df/dw, while optimizers like Adam and Shampoo use the loss gradient dL/dw.
I’ve always wondered which is better, since using the loss gradient with EMA might cause loss spikes later in training.

This paper studies why Adam occasionally causes loss spikes, which is attributed to the edge of stability phenomenon. As seen from the figure, once hitting EOS (see b) a loss spike is triggered. An interesting experimental report!

arxiv.org/abs/2506.04805

I'm looking at ICLR submissions and I've noticed a significant number of papers related to Muon.

「スパース」という単語が関連する分野横断型のワークショップやると，お互いでどういう議論になるか興味がある．．．

学習ダイナミクスや暗黙的バイアスの観点から嬉しいスパース構造と，GPUを用いた行列積にとって嬉しいスパース構造と，脳が持っているスパース構造，どのくらいオーバーラップがあるのですかね🤔GPUを用いた行列積にとって嬉しいスパース行列の構造は複数パターン知られていますが，その学習理論や神経科学との接続はあまり聞かず，ただHPCの人も他分野に興味があるので，関連しそうな文献に引用は飛ばしつつも，あと一歩で行き詰まっている印象？

YouTube video by Chopin Institute KEVIN CHEN – first round (19th Chopin Competition, Warsaw)

And he is the genious.
www.youtube.com/watch?v=iZAp...

YouTube video by Chopin Institute SHIORI KUWAHARA – first round (19th Chopin Competition, Warsaw)

Everyone is giving a wonderful performance, but among the Japanese pianists, I'm particularly fond of Ushida-kun and Kuwahara-san.

www.youtube.com/watch?v=SPS4...
www.youtube.com/watch?v=DaY6...

YouTube video by Chopin Institute ERIC LU – first round (19th Chopin Competition, Warsaw)

I’m watching Chopin Competition.
I’m so surprised that he is using an office chair while I’m deeply impressed by his performance.

m.youtube.com/watch?v=fDsg...

二つの全く違うところでやっていた違うテーマ両方とオーバーラップが大きい研究が出てくると，精神的ダメージが大きいのですが，テーマ設定が安易すぎたのかもしれない

いいテーマがないか放浪する旅をします🧳

ここ2~3ヶ月考えていた２つのテーマに極めて近い論文が，昨日同時に２本出てしまい，研究テーマが消滅してしまった．．．早めに切り替えて，いい研究テーマをゼロから考え直さないといけないですね．．．

PINN，全然知らなかったのですが，最適化&Scalingの対象として面白そう．

Not all scaling laws are nice power laws. This month’s blog post: Zipf’s law in next-token prediction and why Adam (ok, sign descent) scales better to large vocab sizes than gradient descent: francisbach.com/scaling-laws...

I've made some small updates to the 'awesome list' for second-order optimization I made two years ago. It looks like Muon related works and the applications to PINNs have really taken off in the last couple of years.
github.com/riverstone49...

Fluid dynamics might serve as an interesting new benchmark for second-order optimization

Discovery of Unstable Singularities Whether singularities can form in fluids remains a foundational unanswered question in mathematics. This phenomenon occurs when solutions to governing equations, such as the 3D Euler equations, develo...

I don’t know anything about fluid dynamics, but I came across a paper that seemed to say that second-order optimization is key when using the power of neural networks to solve the Navier–Stokes equations. If so, there’s something romantic about that.
arxiv.org/abs/2509.14185

This is not OK.

I don't submit often to NeurIPS, but I reviewed papers for this conference almost every year. As a reviewer, why would I spend time trying to give a fair opinion on papers if it's what happens in the end???

ありがとうございます！

ありがとうございます！また議論できたら幸いです🙏

キーワードとしては，「自然勾配法，Muon，分散学習/継続学習，GPU」周りで出しました．まずは，最適化に関して，最近読んでいるこの三冊の書籍を組み合わせたような論文を一本，出せるよう頑張ります．

ACT-Xに採択されました．引き続き，ニューラルネットワークの最適化について，深い理解を得られるよう研究していきます😁
www.jst.go.jp/kisoken/act-...

Benchmarking Optimizers for Large Language Model Pretraining The recent development of Large Language Models (LLMs) has been accompanied by an effervescence of novel ideas and methods to better optimize the loss of deep learning models. Claims from those method...

When a paper has more than 40 figures, I can really feel the author’s dedication just by looking at it - it’s energizing
arxiv.org/abs/2509.01440

『神経回路網の数理』が英語に翻訳されていないことを考えると，一章ずつ現在の深層学習に対応づけていき，『現代版　神経回路網の数理』を執筆して英語化するプロジェクトとかあったら面白そうですし，初志貫徹の意味では，そのサポートになるような何かをどこかでやりたい．

私が情報系の進路を選んだ一番のきっかけは，「甘利先生の『神経回路網の数理』を現代的なモデルと大規模計算機で実験検証をしてみたい」なのですが，「神経回路網の数理」を久しぶりに開いて読んでみたら，今ちょうどやっていたモデルの簡約化＆解析と実質同じことが書かれていたことに気がつき驚いている．