Olivier Chafik

`server`: streaming of tool calls and thoughts when `--jinja` is on by ochafik · Pull Request #12379 · ggml-org/llama.cpp This PR is still WIP (see todos at the bottom) but welcoming early feedback / testing Support streaming of tool calls in OpenAI format Improve handling of thinking model (DeepSeek R1 Distills, QwQ...

llama.cpp streaming support for tool calling & thoughts was just merged: please test & report any issues 😅

github.com/ggml-org/lla...

#llamacpp

25.05.2025 11:24 — 👍 0    🔁 0    💬 0    📌 0

Runs anywhere (incl. Raspberry Pi 5).
On a Mac:

brew install llama.cpp
llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M

Still fresh / lots of bugs to discover: feedback welcome!

Shoot out to @ggerganov and @ngxson for the patient reviews and general amazing work!

EOT🧵.

01.02.2025 13:45 — 👍 2    🔁 0    💬 0    📌 0

Tool call support (generic + native for Llama, Functionary, Hermes, Mistral, Firefunction, DeepSeek) w/ lazy grammars by ochafik · Pull Request #9639 · ggerganov/llama.cpp This supersedes #6389 (now using a fully C++ approach), #5695 (first attempt at supporting Functionary) and #9592 (more recent Python wrapper). Which models are supported (in their native style)? W...

Llama.cpp now supports tool calling (OpenAI-compatible)

github.com/ggerganov/ll...

On top of generic support for *all* models, it supports 8+ models’ native formats:
- Llama 3.x
- Functionary 3
- Hermes 2/3
- Qwen 2.5
- Mistral Nemo
- Firefunction 3
- DeepSeek R1

🧵 #llamacpp

01.02.2025 13:45 — 👍 5    🔁 1    💬 1    📌 0

Shout out to @ggerganov and the amazing contributors to his llama.cpp adventure for creating such a welcoming and technically thrilling project. One of the most rewarding places to invest hobby time in :-)

github.com/ggerganov/ll...

🧵 4/4

04.12.2024 16:40 — 👍 2    🔁 0    💬 0    📌 0

llama.cpp/grammars at master · ggerganov/llama.cpp LLM inference in C/C++. Contribute to ggerganov/llama.cpp development by creating an account on GitHub.

Note that llama.cpp already has best-in-class JSON schema constraints support, which some already use for tool calling / which my PR builds on (it's been a fun year of PRs!):

github.com/ggerganov/ll...

🧵 3/4

04.12.2024 16:40 — 👍 1    🔁 0    💬 1    📌 0

Tool call support (Llama 3.x, Functionary v3, Hermes 2 Pro, Mistral Nemo, generic) w/ lazy grammars & minimalist Jinja engine by ochafik · Pull Request #9639 · ggerganov/llama.cpp This supersedes #6389 (now using a fully C++ approach), #5695 (first attempt at supporting Functionary) and #9592 (more recent Python wrapper). Background It tackles two main problems related to to...

Forked this off my PR that brings fully-grammar constrained tool call to *all* models (with native prompting style for a few of them):

github.com/ggerganov/ll...

🧵 2/4

04.12.2024 16:40 — 👍 0    🔁 0    💬 1    📌 0

GitHub - google/minja: A minimalistic C++ Jinja templating engine for LLM chat templates A minimalistic C++ Jinja templating engine for LLM chat templates - google/minja

Universal llama.cpp tool call is coming: I've just released Minja, a minimalistic Jinja template engine reimplementation in C++ for LLM chat templates:

github.com/google/minja
(*not an official Google product*)

#LLM #AI #EdgeAI #OSS

🧵 1/4

04.12.2024 16:40 — 👍 6    🔁 2    💬 1    📌 0

llama.cpp/grammars at master · ggerganov/llama.cpp LLM inference in C/C++. Contribute to ggerganov/llama.cpp development by creating an account on GitHub.

To notch it up one bit, you can also specify your own JSON schema (to, say, a list of at between 5 and 10 strings, each conforming to a specific regexp), we've got one of the best support out there

github.com/ggerganov/ll...

04.12.2024 16:34 — 👍 2    🔁 0    💬 0    📌 0

A Gyroid model generated by Manifold's SDF LevelSet function.

#Manifold v3.0 is out! This is a huge release - we have removed *all* required dependencies.

Our npm package is half the size and twice the speed. Our #SDF LevelSet is much faster and higher quality.
And so much more: github.com/elalish/mani...

18.11.2024 18:58 — 👍 9    🔁 5    💬 2    📌 0