luke.shaw@ironarray.io

📢🚨Blosc2 3.11.0 Released! 🚨 📢

Blosc2 is part of the NumPy array library ecosystem: numpy.org

In v3.11, Blosc2 has become even more flexible; all functions can accept basically any array object:

𝚋𝚕𝚘𝚜𝚌𝟸.𝚖𝚊𝚝𝚖𝚞𝚕(𝙰, 𝙱)

for 𝙰, 𝙱 PyTorch, Jax, Tensorflow, Zarr... arrays!

#Compute
#Tensor
#PyTorch

💊IronPill 2💊
See how Blosc2 powers heavy-duty linear algebra (100GB!) workflows
⚡1.5-2x faster than PyTorch + h5py!
🧱 optimised chunking for your cache hierarchy
🐍 one line syntax 𝚋𝚕𝚘𝚜𝚌𝟸.𝚖𝚊𝚝𝚖𝚞𝚕(𝙰, 𝙱, 𝚞𝚛𝚕𝚙𝚊𝚝𝚑='𝚘𝚞𝚝.𝚋𝟸𝚗𝚍')

See blog here: ironarray.io/blog/la-blosc

#Blosc2
#Data
#LinearAlgebra

💊 IronPill 1💊
Blosc2 calculates Fourier approximations:
⚡5x faster than NumPy
👣 fraction of the memory footprint
🐍 one-line syntax 𝚜𝚞𝚖(𝚊 * 𝚌𝚘𝚜(𝚝) + 𝚋 * 𝚜𝚒𝚗(𝚝), 𝚊𝚡𝚒𝚜=𝟷)

Full notebook: github.com/Blosc/python...

(inspiration: towardsdatascience.com/numexpr-the-...)

#Blosc2
#Data
#SignalProcessing

📢Blosc2 3.10.2 Released! 📢

Lazy evaluation for ALL Blosc2 functions in version 3.10!

Blosc2 powers:
- Zarr/HDF5 computations (see image and blosc.org/posts/tensor...)
- Server compuation via Cat2Cloud (demo site here cat2.cloud/demo/)!

#Zarr
#HDF5
#Python

🚨Blosc2 3.9.1🚨

Blosc2 as a mature computation engine >>>

• Array API standard compliance - 70 new functions
• Fast linalg: tensordot 12x 𝐅𝐀𝐒𝐓𝐄𝐑 than NumPy for large arrays !
• Fast calc: Funcs like hypot 10x 𝐅𝐀𝐒𝐓𝐄𝐑 than NumPy!

Blog: ironarray.io/blog/array-api

#Python
#linalg
#computation

🗣️ Announcing Python-Blosc2 3.8.0 🚀

A step closer to compliance with the array-api standard: data-apis.org/array-api!
This is an effort across all array-based libraries so that your code works (e.g. for both blosc2 and NumPy) by simply changing the import statement below!

#arrayapi #Blosc2

Highlights:
✅ C-Blosc2 internal library updated to latest 2.21.2
✅ numexpr version requirement pushed to 2.12.1 to incorporate isnan, isfinite, isinf functions
✅ Better indexing coverage
✅ linspace and arange functions more numerically stable
✅ Many attributes/methods added for array-api compliance

🗣️ Announcing Python-Blosc2 3.8.0 🚀

A step closer to compliance with the array-api standard: data-apis.org/array-api!
This is an effort across all array-based libraries so that your code works (e.g. for both blosc2 and NumPy) by simply changing the import statement below!

#arrayapi #Blosc2

Hats off to the excellent work that @lshaw-ironarray.bsky.social is doing to the @ironarray.io and @blosc.org ecosystems.

Fancy indexing is like a wild animal difficult to tame, but in this particular case he did an outstanding work at it! 👏

Plot on performance of blosc2.concatenate() compared with NumPy

📢 Blosc2 just launched a super-efficient array concatenation feature! 🎉

Combine massive arrays quickly and with minimal memory. If your array chunks are aligned, it's even faster! Perfect for big data tasks.

Check out our blog post: www.blosc.org/posts/blosc2...

Compress Better, Compute Bigger

Caterva2 also supports filtering 🗃️data with fields, via the web client 🌐 or programmatically from Python code 🐍- even the most advanced queries are possible!
This makes it ideal for structured datasets for Data Science 🧪. See 👀 what Caterva2 can do !-> ironarray.io/caterva2.
#DataScience #Python

Did you know Caterva2 has native support for HDF5 files 📂 ? Here at ironArray we are constantly working 🛠️ on the software to offer an improved product to our users - checkout how to use the unfold 📖 command to expose your .h5 files to Caterva2's powerful analysis toolkit 🧰 - > ironarray.io/caterva2

Performance of Blosc2 transposition depending on the chunk size

🗣️New blog entry of Ricardo Sales Piquer, our intern working in linear algebra problems:

Make NDArrays Transposition Fast (and Compressed!) in #Blosc2 🚀

www.blosc.org/posts/optimi...

Great work Ricardo! 🎉

#Compression #LinearAlgebra #Optimization #DataScience

Compress Better, Compute Faster 😀

📝 Just published my blog post as part of the @blosc.org team:

Make NDArrays Transposition Fast (and Compressed!) in #Blosc2 🚀
I dive into optimizing matrix transposition using smart chunk layouts — and how Blosc2 can rival NumPy's speed with compression included.

Compress Better, Compute Faster 😄

Cat2Cloud Video 5!
We connect the Single Source of Truth (SSoT) concept to lazy expressions and Cat2Cloud's powerful append.
We also see how to seamlessly switch between notebooks and the web client.
Full video : ironarray.wistia.com/medias/4uept...
Find out more: ironarray.io/cat2cloud

Our **4th video** on **Cat2Cloud** is now live! 🎉

**Optimize your data analysis pipelines** using **lazy expressions** in Jupyter Notebooks. 📊✨

Seamlessly combine **local 🖥️ and remote 🌐 data processing** 💾⚡

👉 Full video ironarray.io/cat2cloud

Nice intro to using lazy expressions in Cat2Cloud SaaS by our own Luke Shaw 🥳

🚀 Compute Bigger with Cat2Cloud Compression! 🗜️

In this third video, we explore lazy expressions 😴 using the Cat2Cloud web client. 🌐

💡 Lazy expressions decompress and compute data only when needed, saving resources!

👉 Try it: cat2.cloud/demo

🔗 ironarray.io/cat2cloud 👀
#compression #datascience

Dealing with remote Blosc2 datasets has never been easier 😊
Follow the video interactively with: github.com/ironArray/Ca...
Welcome to the team, Luke!

Second cat2cloud intro video! 🥳
Optimise file transfer ⚡ and access for server-hosted data 📅 with cat2cloud's compression-first framework. Minimise transfer times/storage requirements!
More -> ironarray.io/cat2cloud!

Computing with arrays way larger than RAM, but still with excellent performance.

Is it possible to compute with arrays that are 100x larger than memory and still achieve good performance? 🤯

With the new compute engine in Python-Blosc2, you can! 😊 Check out our blog post for more details: ironarray.io/blog/compute...

Compress Better, Compute Bigger!

🚀 Blosc2 is blasting groundbreaking compression and computing power straight to the cloud! 🔥💡

Transposing a compressed array with Blosc2

📣 Announcing the release of Python-Blosc2 3.3.0:

🔄 New blosc2.transpose()
⚡️ New fast path for NDArray.slice() - up to 40x speedup
🔧 NDArray.slice() now preserves compression params
📝 Improved documentation

Thanks to all contributors!

github.com/Blosc/python...

Compress Better, Compute Bigger! 😀

First video explaining the roots concept, and how to upload and download data

🗣️ After 15 years delivering Blosc/Blosc2, we are glad to announce the beta program for Cat2Cloud:🤟

- 🗜️ Access to the best compression algorithms available
- 🔄 Share complex data securely and effortlessly
- ⚡ Perform advanced computations directly in the cloud

...and more!

ironarray.io/cat2cloud

blosc2.jit can accelerate computations a lot, and very easily.

Did you know that you can use the @blosc2.jit decorator to accelerate your computations with NumPy by 10x and more by just adding a single line to your function?

No need to add loops manually, just add @blosc2.jit and it will do its magic!

Compress Better, Compute Bigger!

📢 We are happy to announce the release of Python-Blosc2 3.2.0!

This release introduces the new blosc2.matmul() function. We have written a nice blog: www.blosc.org/posts/optimi...

Lastly, we are excited to introduce WASM32 wheels for the first time. More info at: github.com/Blosc/python...

Enjoy!

We've poured our hearts into creating comprehensive documentation for Python-Blosc2. Explore the latest resources at:

- www.blosc.org/python-blosc2/

Your feedback is invaluable in shaping these materials to better serve the community. Let us know what you think!

Compress Better, Compute Bigger!

The plot shows the performance of the new Blosc2 compute capabilities. It can be more than 10x faster than NumPy and numba.

🔥 Blosc2 3.2.1: NumPy's best friend! 🔥 blosc2.jit + NumPy array interface = blazing fast, direct compressed data crunching!

See how: github.com/Blosc/python-blosc2/blob/main/examples/ndarray/jit-numpy-funcs.py

#DataScience #Python #Compression

Compress Better, Compute Bigger!

Up to 75x speedups of Blosc2 vs NumPy and Numba when computing with generic NumPy functions

📢 🔥 Article on Blosc2: Compute with TB-sized datasets on your own hardware!

🚀 Outperforms NumPy by 10x ~ 100x for large computations
🐍 Integrates seamlessly with the Python data science ecosystem
💻 Works both in-memory and on-disk with minimal performance differences

ironarray.io/blog/compute...

Performance of Blosc2 vs Dask+Zarr+Blosc for a complex math calculation.

Blosc compression helped fighting memory bottlenecks for 15 years now! 🎂
See how Dask + Zarr benefits from it. But vertical integration between compression and the compute engine in newest Python-Blosc2 makes a big difference in terms of speed ⚡ and scalability 🤟

#Compress better, #Compute bigger!