Michael Horrell

Some pitfalls in vibe-coding seen along the way:

- Unchecked, cursor would have deleted all code calculating gradients and hessians, breaking everything.

- The first version completely skipped the obvious get/set_extra_state PyTorch methods, creating problems. Possibly these are newer methods?

Credit to PyTorch, XGB and LightGBM btw. This was surprisingly easy to accomplish: (1) serialize GBMs to string (2) store the string in PyTorch state_dict.

Release v0.5.2 · mthorrell/gbnet What's Changed adjusted state storage by @mthorrell in #86 Update setup.py by @mthorrell in #87 Full Changelog: v0.5.1...v0.5.2

github.com/mthorrell/gb...

Speaking at @scipyconf.bsky.social was a blast!

Based on questions from the audience, I released an update to GBNet that enables saving a loading models to disk.

One step closer to V1.0

Just published in JOSS: 'GBNet: Gradient Boosting packages integrated into PyTorch' https://doi.org/10.21105/joss.08047

🔥 Speaker Spotlight: Michael Horrell

Don't miss @horrellmt.bsky.social at #SciPy2025 for "GBNet: Gradient Boosting packages integrated into PyTorch." 🧠⚙️

Discover how GBNet bridges powerful boosting methods with deep learning workflows.

🔗 scipy2025.scipy.org

GitHub - mthorrell/gbnet: Gradient Boosting Modules for pytorch Gradient Boosting Modules for pytorch. Contribute to mthorrell/gbnet development by creating an account on GitHub.

Find the package here: github.com/mthorrell/gb...

In my talk I plan to cover applications in Forecasting, Ordinal Regression, Mixture of Experts (MoE), even Matrix Factorization, all with XGBoost or LightGBM forming major model components.

If you're going to be at #SciPy2025, come find me, I'd love to chat!

If you're a user of XGBoost or LightGBM and want to do more with those already amazing tools, then take a look at GBNet.

GBNet connects XGBoost and LightGBM to PyTorch to enable a wide range of new applications for gradient boosting models.

🚀 #SciPy2025 is just under two weeks away, and I’ll be there!

🎙️ Talk: “GBNet: Gradient Boosting Packages Integrated into PyTorch”
🗓️ Wed July 9, 11:25 AM (Room 315)

I'll be speaking about GBNet, an Open Source package I maintain.

Side note: if you extend this case to use different numbers of observations and different variances, you get actually a pretty nice looking matplotlib plot

Bayesian modeling provides a nice test bed for this

Eg: When finding a mean, how much to weight the prior (call it Model 1) vs the sample average (Model 2)? Some bayesian math gives the optimal answer given # of obs.

A bit of GBNet & PyTorch empirically derives the same answer almost exactly.

The simplest version of this is:

- Model1 is some fancy model that may have some bad edge case behavior
- Model2 is a basic average

A straightforward GBNet PyTorch Module will allow one to fit these coefficients.

Next application for GBNet is basically a data-aware model averaging or a mixture of experts type of analysis.

Situation: you have several models with predictions

Q: Is there a data-driven way to combine them? And, for convenience, can I use XGBoost to do find the right averaging coefficients?

This feels a lot like normal embeddings, though unlike PyTorch embeddings, these embeddings will be fixed at 0 unless there is an empirical reason to have them be non-zero.

GBNet calls XGBoost/LightGBM under the hood. This means you can bring native XGB/LGBM features to PyTorch with little effort.

Categorical splitting is one interesting feature to play with using GBNet. To scratch the surface, I fit a basic Word2Vec model using XGBoost for categorical splitting.

Release v0.5.0 · mthorrell/gbnet What's Changed added uncertainty ests by @mthorrell in #73 Full Changelog: v0.4.0...v0.5.0

If you use Meta's Prophet package, give GBNet a look. It might provide significant lift with a one line code change.

github.com/mthorrell/gb...

Just released GBNet V0.5.0

Beyond some usability improvements, the uncertainty estimation for the Forecasting module got merged in. Now GBNet forecasting is:

✅ Faster
✅ More accurate than Prophet
✅ Provides uncertainty estimates
✅ Supports changepoints

One benefit to speeding up model fitting code 5X is that you can use that saved time for other things.

Adding conf intervals for gbnet forecasting module, I can do train/validation holdout for this and still be 3-4X faster.

Trying to get 80% test coverage:
New method: 76% avg
Prophet: 55% avg

So it's continually a nice surprise that stuff like this kinda just works.

I asked GBNet for a second prediction output. I slapped on torch.nn.GaussianNLLLoss and out comes a variance estimate that is well calibrated.

Next is adding estimates of uncertainty. I _think_ I can get away with having the GBM output a standard deviation and then I just use a gaussian likelihood (rather than pure MSE), but we will see.

It's worth mentioning that changepoints, while they do improve performance, do not seem strictly necessary for many datasets. When the core model is Trend + GBM(t), the GBM by itself truly can handle a lot of non-stationary behavior and still give great performance.

Just merged changepoints into the forecasting sub-module of GBNet and released V0.4.0.

Default forecast performance improved by 20% and achieved a 5X speedup. Using the random training, random horizon benchmark, now 9 of 9 example datasets have better performance with GBNet compared to Prophet.

Explaining the equation: The slope of the (now non-linear) trend-line is a base slope + adjustments based on points in time. Those adjustments are determined by GBDT AND the outputs of the GBDT need to be summed to produce a continuous line. You'd have to use GBNet to do something like this.

Still working on changepoints. Several methods work but don't improve benchmarks. When your model is Trend + XGBoost, XGB can just handle a lot of non-stationarity.

Most promising method so far (see plot) asks GBDT to fit and find the changepoints. Another cool application of GBNet (see equation).

Claude gets half credit. It added changepoints for the PyTorch trend option but skipped it for the GBLinear option.

Unfortunately it's back to the drawing board. PyTorch changepoints fit too slowly and GBLinear, I now realize, can't actually turn them off. It does work though!

Next, I'll add changepoints to the Forecasting model for GBNet. With this feature, Prophet's primary prediction functionality will be available in GBNet.

I wonder how much the LLMs will be able to one-shot it. According to the Prophet paper it's just a broken stick regression with lasso penalty.

docs: gbnet.readthedocs.io/en/stable/

Release v0.3.0 · mthorrell/gbnet What's Changed added gblinear by @mthorrell in #57 added general class by @mthorrell in #59 Update forecast w gblinear by @mthorrell in #60 Docs by @mthorrell in #61 Update README.md by @mthorrell...

github.com/mthorrell/gb...

Just released v0.3.0 of GBNet.

New features:
- ReadTheDocs website
- GBLinear
- GBLinear integration into forecasting model (10X faster, 10% improvement in predictions)
- Python class refactoring