llama-crab Run local GGUF models from your applications

llama-crab is a Rust SDK, installable HTTP server, Tauri plugin, and TypeScript client for running local llama.cpp models. Current version: 0.1.8 (Rust crates and TypeScript packages released in lockstep). MSRV: 1.88.

Start with the surface you plan to use:

Get started GitHub

Pick the integration that matches your stack

Rust SDK

Embed local inference directly in Rust applications. Load GGUF models, run text completion, chat, embeddings, structured output, multimodal vision, and (in 0.1.8) resolve Hugging Face repository IDs through the hf-hub feature.

HTTP server

Run llama-crab-server to expose local models behind OpenAI-compatible routes: completions, chat, embeddings, reranking, tokenization, and SSE streaming. The mtmd and hf-hub features are opt-in.

Tauri plugin

Ship local inference inside Tauri v2 desktop applications with tauri-plugin-llama-crab and the @llama-crab/tauri TypeScript client. The plugin enables hf-hub by default and exposes multimodal (vision) chat behind the mtmd cargo feature.

TypeScript packages

Use @llama-crab/core for shared OpenAI-like contracts, request mappers, and error classes, and @llama-crab/tauri for the Tauri IPC client. Both ship ESM, CJS, and type declarations at version 0.1.8.

Built for production workloads

Five products, one runtime

llama-crab (Rust SDK), llama-crab-server (HTTP server), tauri-plugin-llama-crab (Tauri v2 plugin), @llama-crab/core (TS contracts), and @llama-crab/tauri (TS client). All are at version 0.1.8.

CPU and GPU backends

Pick a Cargo feature for your target: openmp for CPU, metal for Apple Silicon, cuda for NVIDIA, vulkan, rocm, or opencl for AMD/cross-vendor, and kleidiai for Arm. mtmd enables multimodal vision.

OpenAI-compatible API

Local models speak the same shape as OpenAI: 12 routes including /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/rerank (with three aliases), and extras for tokenization and model listing. SSE streaming follows the OpenAI event format.

Mobile presets

LowRam, Balanced, and GpuMax presets match device class so you can ship the same API across phones, laptops, and workstations. The server mirrors the presets through --mobile-preset low-ram | balanced | gpu-max.

Get started in minutes

Install the crate

Add llama-crab to your Rust manifest, install the C++ toolchain and CMake, then load a GGUF model in a few lines of Rust. MSRV 1.88.

Run your first completion

Create a small Rust binary, point it at a quantized GGUF model, and run a cargo run --release to see generated text.

Tune for performance

Set context size, batch size, GPU offload, and threads. Start with the smallest model that exercises your workload and tune one variable at a time.

Troubleshooting

Common failure modes for the server, Tauri plugin, and TypeScript client, with the exact flags and configurations that resolve them.