Dan

Lollo the Bichon Friseé in christmas outfit.

24.12.2024 07:38 — 👍 0    🔁 0    💬 0    📌 0

Release v0.4.4 · DanEngelbrecht/golongtail FIXED fix(s3): use HeadObject for checking if blob exists bergemalm ADDED New option to control the number of worker thread in remote stores to avoid overflowing the network connection. --remote-w...

New maintenance releases of longtail and golongtail

golongtail 0.4.4
github.com/DanEngelbrec...

longtail 0.4.3 github.com/DanEngelbrec...

23.11.2024 13:20 — 👍 0    🔁 0    💬 0    📌 0

If you are using longtail as a lib you need to switch from `Longtail_ChangeVersion()` to `Longtail_ChangeVersion2()` and use `Longtail_CreateConcurrentChunkWriteAPI`. It is that you drop the LRU block store layer is becomes redundant and only adds memory overhead.
Very happy with how this turned out

03.01.2024 23:22 — 👍 0    🔁 0    💬 0    📌 0

Turns out the new strategy is significantly simpler, uses less memory, reduces file I/O, cuts down on CPU usage and drastically reduces wall time to write a new version when it contains files larger than a single block.

03.01.2024 23:21 — 👍 0    🔁 0    💬 1    📌 0

The implementation was focused sequential writes of files that caused huge amount of extra I/O, memory and CPU work to be done.
Now, realizing that I don't *need* to write big files from start to finish, but instead I can write the parts in any order, I got to work implementing a new strategy.

03.01.2024 23:20 — 👍 0    🔁 0    💬 1    📌 0

Just before the holiday break at work I noticed that writing a version from an archive containing files larger than a single block was **very** slow - like, slow as in "this can't be right".
But it was.

03.01.2024 23:20 — 👍 0    🔁 0    💬 1    📌 0

Image showing memory usage, and wall time of speed ups of the new longtail version. 2.7 -> 3.25 x less memory, 2.2 -> 14 x faster

Happy new year all.
I finally got some time and energy to poke around with longtail.
New release includes `Longtail_ChangeVersion2()` used by default used by the cmdline tools.
Massive speed-ups for assets larger than one block.
Details in thread
github.com/DanEngelbrec...
github.com/DanEngelbrec...

03.01.2024 23:18 — 👍 0    🔁 0    💬 1    📌 0

Just release 0.4.0 of both longtail and golongtail.
Major point for both is MacOS Arm64 builds.

github.com/DanEngelbrec...

github.com/DanEngelbrec...

08.09.2023 19:51 — 👍 1    🔁 0    💬 0    📌 0