atomk-hermes-skills/skills/mlops/llm-fine-tuning/SKILL.md

name, description, version, author, license, metadata

name	description	version	author	license	metadata
llm-fine-tuning	LLM fine-tuning: framework comparison (Axolotl, TRL, Unsloth) and method selection (SFT, DPO, PPO, GRPO).	1.0.0	Hermes Agent	MIT	hermes tags related_skills fine-tuning llm axolotl trl unsloth lora qlora dpo ppo grpo rlhf sft training huggingface-hub llm-inference

LLM Fine-Tuning

Three frameworks for fine-tuning LLMs, each optimized for different use cases. Choose based on your training objective, then load the detailed reference for that framework.

Choosing a Framework

Need	Framework	Why
Production-grade multi-method training	Axolotl	Most mature, YAML-driven, 100+ model presets, LoRA/QLoRA/DPO/KTO/ORPO/GRPO
Reinforcement learning from human feedback	TRL	Native SFT/DPO/PPO/GRPO, HuggingFace-native, research-friendly
Fastest training on limited hardware	Unsloth	2-5x faster, 50-80% less memory, great for LoRA/QLoRA on single GPU

Choosing a Method

Method	Use When	Training Data
SFT (Supervised Fine-Tuning)	Instruction tuning, domain adaptation	Prompt-completion pairs
DPO (Direct Preference Optimization)	Align with preferences, no reward model needed	Chosen/rejected pairs
PPO (Proximal Policy Optimization)	RLHF with online reward model	Prompt + reward model
GRPO (Group Relative Policy Optimization)	Rule-based rewards, no value model needed	Prompt + rule-based reward
KTO/ORPO	Simpler alignment, no pairwise preferences	Binary good/bad signals
LoRA/QLoRA	Parameter-efficient, limited GPU VRAM	(applies to any method above)

Section 1: Axolotl — YAML-Driven Multi-Method Training

Production-grade fine-tuning framework. YAML configs, 100+ model presets, LoRA/QLoRA, DPO/KTO/ORPO/GRPO, multimodal support.

When to use: Production training jobs, multi-method experiments, need reproducibility, working with large models.

Key features:

• YAML-driven configuration (no Python scripts needed)
• 100+ model presets (Llama, Mistral, Qwen, Gemma, etc.)
• DeepSpeed FSDP integration for multi-GPU
• LoRA/QLoRA/DPO/KTO/ORPO/GRPO support
• Multimodal fine-tuning (vision-language models)
• Cloud backends (Modal, RunPod)

Quick start:

pip install axolotl
# Create a YAML config, then:
accelerate launch -m axolotl.cli.train config.yaml

Key config pattern (FSDP v2):

fsdp_version: 2
fsdp_config:
  offload_params: true
  state_dict_type: FULL_STATE_DICT
  auto_wrap_policy: TRANSFORMER_BASED_WRAP

Pitfalls:

• context_parallel_size must divide total GPUs
• save_compressed: true reduces disk 40%, compatible with vLLM
• Integration code need not go in integrations/ folder
• Handle both single-example and batched data shapes in custom collators

See: references/axolotl.md for full config reference, API docs, and dataset formats.

Section 2: TRL — Transformer Reinforcement Learning

HuggingFace-native post-training library. SFT, DPO, PPO, GRPO, and reward model training.

When to use: RLHF workflows, preference alignment, research experiments, need tight HuggingFace integration.

Key features:

• SFTTrainer — supervised instruction tuning
• DPOTrainer — direct preference optimization
• PPOTrainer — proximal policy optimization (classic RLHF)
• GRPOTrainer — group relative policy optimization (no value model)
• Reward model training and online RL methods
• Works with any HuggingFace model + PEFT for LoRA

Quick start (SFT):

from trl import SFTTrainer
trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,
)
trainer.train()

Quick start (DPO):

from trl import DPOTrainer, DPOConfig
config = DPOConfig(output_dir="model-dpo", beta=0.1)
trainer = DPOTrainer(
    model=model, args=config,
    train_dataset=preference_dataset,
    processing_class=tokenizer
)
trainer.train()

Pitfalls:

• DPO beta controls alignment strength — too high reduces diversity
• PPO requires a reward model + reference model (memory-intensive)
• GRPO eliminates the value model but needs careful reward normalization
• Always use processing_class= instead of deprecated tokenizer=

See: references/trl-fine-tuning.md for full method references and templates/ for starter scripts.

Section 3: Unsloth — Fast & Memory-Efficient Fine-Tuning

2-5x faster training with 50-80% less memory. Optimized for LoRA/QLoRA on consumer GPUs.

When to use: Single-GPU training, limited VRAM (4-24GB), rapid iteration, learning/experimentation.

Key features:

• 2-5x faster than standard HuggingFace training
• 50-80% less VRAM usage
• Auto-selects optimal LoRA/QLoRA parameters
• Supports Llama, Mistral, Gemma, Qwen architectures
• Drop-in replacement: same HuggingFace Trainer API

Quick start:

from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b-bnb-4bit",
    max_seq_length=2048,
    load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
    model, r=16,
    lora_alpha=16, lora_dropout=0,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

Pitfalls:

• Only supports specific architectures (Llama, Mistral, Gemma, Qwen)
• May lag behind upstream Transformers version
• Best for LoRA; full fine-tuning not the primary target

See: references/unsloth.md for full documentation.

Comparison Table

Feature	Axolotl	TRL	Unsloth
SFT	✅	✅	✅
DPO	✅	✅	❌
PPO/GRPO	✅	✅	❌
LoRA/QLoRA	✅	✅ (via PEFT)	✅ (native)
Full fine-tune	✅	✅	⚠️ Limited
Multi-GPU	✅ DeepSpeed	✅ Accelerate	⚠️ Limited
YAML config	✅	❌ Python	❌ Python
Speed	Baseline	Baseline	2-5x faster
Memory	Baseline	Baseline	50-80% less
Cloud backends	Modal, RunPod	Any	Any

Workflow: Which Path?

1. Need the fastest iteration on limited hardware? → Start with Unsloth
2. Need RLHF / preference alignment? → Use TRL (or Axolotl's TRL backend)
3. Need production reproducibility + multi-method? → Use Axolotl
4. Training complete? → Export to GGUF (see llm-inference skill) or serve with vLLM