GGUF / Local LLM Agent Plugin (`ovos-gguf-plugin`)¶

In a nutshell

This plugin runs an AI language model entirely on your own device. No internet, no accounts, and nothing sent to a company's servers. This is ideal when privacy or offline use matters. ("GGUF" is the file format these downloadable models come in.) It can chat, summarize, translate, detect languages, and more. See AI Agents & Personas for the bigger picture and the Glossary for unfamiliar terms.

ovos-gguf-plugin runs local GGUF models through llama-cpp-python. It needs no API keys and no network, so it suits offline, air-gapped, or privacy-sensitive deployments. It provides a chat engine, a summarizer, translation and language-detection, a text-embeddings plugin, and a dialog transformer.

Models load from a Hugging Face Hub repository (downloaded on first use) or from a local .gguf file path. Each engine loads its own llama_cpp.Llama instance. You can share one Llama between engines by passing gguf_engine= in code.

Install: pip install --pre ovos-gguf-plugin

GitHub: OpenVoiceOS/ovos-gguf-plugin

Plugins Overview¶

Entry point	Plugin name	Class	Source
`opm.agents.chat`	`ovos-chat-gguf-plugin`	`GGUFChatEngine`	`ovos_gguf_plugin/chat.py`
`opm.agents.summarizer`	`ovos-summarizer-gguf-plugin`	`GGUFSummarizer`	`ovos_gguf_plugin/summarizer.py`
`opm.lang.translate`	`ovos-translate-gguf-plugin`	`GGUFTextTranslator`	`ovos_gguf_plugin/translate.py`
`opm.lang.detect`	`ovos-lang-detect-gguf-plugin`	`GGUFTextLangDetector`	`ovos_gguf_plugin/translate.py`
`opm.transformer.dialog`	`ovos-dialog-transformer-gguf-plugin`	`GGUFDialogTransformer`	`ovos_gguf_plugin/dialog_transformers.py`
`opm.embeddings.text`	`ovos-gguf-embeddings-plugin`	`GGUFEmbeddings`	`ovos_gguf_plugin/embeddings.py`

Scope

This plugin does not ship coref / reranker / extractive-QA / NLI / yes-no / memory engines, a chat-summarizer, or an utterance transformer. For reranking, install a reranker plugin such as ovos-flashrank-reranker-plugin (no opm.agents.reranker implementation ships yet, see Agent Plugins).

Common Configuration Keys¶

Model loading happens in GGUFChatEngine, which the other engines delegate to.

Key	Type	Default	Description
`model`	`str`	required	Hugging Face repo id (`"owner/repo-name-GGUF"`) or absolute path to a `.gguf` file.
`remote_filename`	`str`	`*Q4_K_M.gguf`	Glob selecting the GGUF file from a Hub repo (ignored for local-file `model`).
`n_gpu_layers`	`int`	`0`	Layers offloaded to GPU. `0` means CPU only. `-1` means all layers on GPU.
`chat_format`	`str`	`null`	`llama-cpp-python` chat template name (auto-detected if unset).
`max_tokens`	`int`	`null`	Maximum tokens in the completion (`null` = model/llama.cpp default).
`verbose`	`bool`	`true`	Pass-through to `llama_cpp.Llama`.
`system_prompt`	`str`	`null`	Default system prompt.
`allow_system_prompts`	`bool`	`false`	When `true`, caller system messages are merged with the configured prompt. When `false`, they are stripped.

A model value that is an existing file path is loaded with Llama(model_path=...). Otherwise it is treated as a Hub repo id and loaded with Llama.from_pretrained(repo_id=..., filename=...).

Translation and language detection override two defaults

ovos-translate-gguf-plugin and ovos-lang-detect-gguf-plugin set n_gpu_layers to -1 (full GPU offload) instead of 0. The translator also ships a working default model, TheBloke/TowerInstruct-7B-v0.1-GGUF, so model is not required there — only the language detector still requires it explicitly.

Minimal configuration (Hub model)¶

{
  "model": "microsoft/Phi-3-mini-4k-instruct-gguf",
  "remote_filename": "*q4.gguf",
  "n_gpu_layers": 0
}

First run downloads the model

The first time a Hub model is used, Llama.from_pretrained downloads the matching GGUF file from Hugging Face and caches it locally. Later runs are instant. For the example above, the quantized file (Phi-3-mini-4k-instruct-q4.gguf) is about 2.2 GiB. Expect anywhere from under a minute to several minutes, depending on your internet connection. Larger models (7B+) commonly run several GiB and take longer. The download only happens once per file. No data is sent anywhere afterward: inference is fully offline.

Local file configuration¶

{
  "model": "/home/user/models/llama-3.1-8b.gguf",
  "n_gpu_layers": 20
}

Chat Engine (`opm.agents.chat`)¶

Class: GGUFChatEngine (ovos_gguf_plugin/chat.py:GGUFChatEngine)

OPM plugin name: ovos-chat-gguf-plugin

Multi-turn conversational LLM using a local GGUF model. Implements continue_chat, stream_tokens, and stream_sentences. It is API-compatible with OpenAIChatEngine for offline use.

{
  "ovos-chat-gguf-plugin": {
    "model": "microsoft/Phi-3-mini-4k-instruct-gguf",
    "remote_filename": "*q4.gguf",
    "n_gpu_layers": 0,
    "system_prompt": "You are a helpful assistant."
  }
}

Offline persona example¶

{
  "name": "Local Phi-3",
  "handlers": ["ovos-chat-gguf-plugin"],
  "ovos-chat-gguf-plugin": {
    "model": "microsoft/Phi-3-mini-4k-instruct-gguf",
    "remote_filename": "*q4.gguf",
    "n_gpu_layers": 0,
    "system_prompt": "You are a concise, helpful voice assistant."
  }
}

Activate by voice: "Chat with Local Phi-3".

Summarizer (`opm.agents.summarizer`)¶

Class: GGUFSummarizer (ovos_gguf_plugin/summarizer.py:GGUFSummarizer)

OPM plugin name: ovos-summarizer-gguf-plugin

Condenses a document into a short summary using a local GGUF model. Delegates generation to a GGUFChatEngine.

The system prompt and user prompt default to the plugin's localized .prompt files. Override with system_prompt and prompt_template (a template with a {content} placeholder).

Key	Type	Default	Description
`system_prompt`	`str`	localized `summarize_system` prompt	Instruction for the summarisation model.
`prompt_template`	`str`	localized `summarize_user` prompt	Template with a `{content}` placeholder.

{
  "ovos-summarizer-gguf-plugin": {
    "model": "/path/to/model.gguf",
    "max_tokens": 256
  }
}

Translation & Language Detection (`opm.lang.translate`, `opm.lang.detect`)¶

Classes: GGUFTextTranslator / GGUFTextLangDetector (ovos_gguf_plugin/translate.py)

OPM plugin names: ovos-translate-gguf-plugin / ovos-lang-detect-gguf-plugin

Translate text between languages or detect a text's language using a local GGUF model.

{
  "language": {
    "translation_module": "ovos-translate-gguf-plugin",
    "detection_module": "ovos-lang-detect-gguf-plugin",
    "ovos-translate-gguf-plugin": {
      "model": "/path/to/model.gguf"
    },
    "ovos-lang-detect-gguf-plugin": {
      "model": "/path/to/model.gguf"
    }
  }
}

Dialog Transformer (`opm.transformer.dialog`)¶

Class: GGUFDialogTransformer (ovos_gguf_plugin/dialog_transformers.py:GGUFDialogTransformer)

OPM plugin name: ovos-dialog-transformer-gguf-plugin

Runs after skill response generation, before TTS synthesis. Rewrites skill responses with a local GGUF model. It only runs when a rewrite_prompt is configured (via config or context["prompt"]). Otherwise it falls back to the original dialog.

Default priority: 10. The system prompt defaults to the localized dialog_transform_system prompt.

{
  "dialog_transformers": {
    "ovos-dialog-transformer-gguf-plugin": {
      "model": "/path/to/model.gguf",
      "rewrite_prompt": "Rewrite in a warm, friendly tone. Remove markdown.",
      "n_gpu_layers": 20
    }
  }
}

Model Selection Guide¶

Use case	Recommended model size	Example
Real-time voice (low latency)	1B to 3B parameters	`Phi-3-mini` Q4_K_M
General purpose	7B to 8B parameters	`Llama-3.1-8B` Q4_K_M
Memory-constrained devices	1B to 1.5B parameters	`Qwen2.5-1.5B` Q4_K_M

Quantization level guide:

Q4_K_M: good balance of quality and speed (recommended default)
Q8_0: higher quality, roughly 2x the memory of Q4
Q2_K: smallest/fastest, lowest quality

GPU Acceleration¶

Set n_gpu_layers to offload transformer layers to a CUDA or Metal GPU:

{
  "model": "/path/to/llama-3.1-8b.gguf",
  "n_gpu_layers": -1
}

-1 offloads all layers (full GPU inference). Values > 0 offload that many layers (partial offload for limited VRAM). 0 = CPU only.

Requires llama-cpp-python to be compiled with CUDA or Metal support:

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall

Source code: OpenVoiceOS/ovos-gguf-plugin.

Read next: LLM Transformers Related: OpenAI-compatible · Agent Engine Types · Personas & PersonaService

GGUF / Local LLM Agent Plugin (ovos-gguf-plugin)¶