atomk-hermes-skills/skills/mlops/llm-inference/SKILL.md

name, description, version, author, license, platforms, metadata

name	description	version	author	license	platforms	metadata
llm-inference	LLM inference: local GGUF inference (llama.cpp) and high-throughput serving (vLLM).	2.0.0	Hermes Agent	MIT	linux macos	hermes tags related_skills llm inference llama-cpp vllm gguf serving deployment huggingface-hub

LLM Inference

Two inference approaches: local GGUF inference for development and vLLM for production serving.

Section 1: llama.cpp — Local GGUF Inference

Run LLMs locally using llama.cpp with GGUF quantized models. Supports CPU, GPU, and mixed inference.

When to use: Local development, testing models, resource-constrained environments, offline inference. Key features: GGUF quantization, CPU/GPU/mixed, OpenAI-compatible server mode, HF Hub model discovery. See: references/llama-cpp.md.

Section 2: vLLM — High-Throughput Serving

Deploy LLMs for production with vLLM. OpenAI API compatibility, PagedAttention, continuous batching, and quantization support.

When to use: Production LLM deployment, high-throughput serving, multi-model serving, needs OpenAI API compatibility. Key features: PagedAttention, continuous batching, tensor parallelism, GPTQ/AWQ quantization, OpenAI API server. See: references/serving-llms-vllm.md.

Choosing Between Them

Need	Tool
Local dev/testing	llama.cpp
Resource-constrained	llama.cpp (with quantization)
Production API serving	vLLM
High throughput needed	vLLM
OpenAI API compatibility	Both (via server modes)
Multi-GPU serving	vLLM (tensor parallelism)