Add mlops/llm-fine-tuning

skills/mlops/llm-fine-tuning/SKILL.md

@@ -0,0 +1,173 @@
+---
+name: llm-fine-tuning
+description: "LLM fine-tuning: framework comparison (Axolotl, TRL, Unsloth) and method selection (SFT, DPO, PPO, GRPO)."
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+metadata:
+  hermes:
+    tags: [fine-tuning, llm, axolotl, trl, unsloth, lora, qlora, dpo, ppo, grpo, rlhf, sft, training]
+    related_skills: [huggingface-hub, llm-inference]
+---
+
+# LLM Fine-Tuning
+
+Three frameworks for fine-tuning LLMs, each optimized for different use cases. Choose based on your training objective, then load the detailed reference for that framework.
+
+## Choosing a Framework
+
+| Need | Framework | Why |
+|------|-----------|-----|
+| Production-grade multi-method training | **Axolotl** | Most mature, YAML-driven, 100+ model presets, LoRA/QLoRA/DPO/KTO/ORPO/GRPO |
+| Reinforcement learning from human feedback | **TRL** | Native SFT/DPO/PPO/GRPO, HuggingFace-native, research-friendly |
+| Fastest training on limited hardware | **Unsloth** | 2-5x faster, 50-80% less memory, great for LoRA/QLoRA on single GPU |
+
+## Choosing a Method
+
+| Method | Use When | Training Data |
+|--------|----------|---------------|
+| **SFT** (Supervised Fine-Tuning) | Instruction tuning, domain adaptation | Prompt-completion pairs |
+| **DPO** (Direct Preference Optimization) | Align with preferences, no reward model needed | Chosen/rejected pairs |
+| **PPO** (Proximal Policy Optimization) | RLHF with online reward model | Prompt + reward model |
+| **GRPO** (Group Relative Policy Optimization) | Rule-based rewards, no value model needed | Prompt + rule-based reward |
+| **KTO/ORPO** | Simpler alignment, no pairwise preferences | Binary good/bad signals |
+| **LoRA/QLoRA** | Parameter-efficient, limited GPU VRAM | (applies to any method above) |
+
+## Section 1: Axolotl — YAML-Driven Multi-Method Training
+
+Production-grade fine-tuning framework. YAML configs, 100+ model presets, LoRA/QLoRA, DPO/KTO/ORPO/GRPO, multimodal support.
+
+**When to use:** Production training jobs, multi-method experiments, need reproducibility, working with large models.
+
+**Key features:**
+- YAML-driven configuration (no Python scripts needed)
+- 100+ model presets (Llama, Mistral, Qwen, Gemma, etc.)
+- DeepSpeed FSDP integration for multi-GPU
+- LoRA/QLoRA/DPO/KTO/ORPO/GRPO support
+- Multimodal fine-tuning (vision-language models)
+- Cloud backends (Modal, RunPod)
+
+**Quick start:**
+```bash
+pip install axolotl
+# Create a YAML config, then:
+accelerate launch -m axolotl.cli.train config.yaml
+```
+
+**Key config pattern (FSDP v2):**
+```yaml
+fsdp_version: 2
+fsdp_config:
+  offload_params: true
+  state_dict_type: FULL_STATE_DICT
+  auto_wrap_policy: TRANSFORMER_BASED_WRAP
+```
+
+**Pitfalls:**
+- `context_parallel_size` must divide total GPUs
+- `save_compressed: true` reduces disk 40%, compatible with vLLM
+- Integration code need not go in `integrations/` folder
+- Handle both single-example and batched data shapes in custom collators
+
+**See:** `references/axolotl.md` for full config reference, API docs, and dataset formats.
+
+## Section 2: TRL — Transformer Reinforcement Learning
+
+HuggingFace-native post-training library. SFT, DPO, PPO, GRPO, and reward model training.
+
+**When to use:** RLHF workflows, preference alignment, research experiments, need tight HuggingFace integration.
+
+**Key features:**
+- `SFTTrainer` — supervised instruction tuning
+- `DPOTrainer` — direct preference optimization
+- `PPOTrainer` — proximal policy optimization (classic RLHF)
+- `GRPOTrainer` — group relative policy optimization (no value model)
+- Reward model training and online RL methods
+- Works with any HuggingFace model + PEFT for LoRA
+
+**Quick start (SFT):**
+```python
+from trl import SFTTrainer
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-0.5B",
+    train_dataset=dataset,
+)
+trainer.train()
+```
+
+**Quick start (DPO):**
+```python
+from trl import DPOTrainer, DPOConfig
+config = DPOConfig(output_dir="model-dpo", beta=0.1)
+trainer = DPOTrainer(
+    model=model, args=config,
+    train_dataset=preference_dataset,
+    processing_class=tokenizer
+)
+trainer.train()
+```
+
+**Pitfalls:**
+- DPO `beta` controls alignment strength — too high reduces diversity
+- PPO requires a reward model + reference model (memory-intensive)
+- GRPO eliminates the value model but needs careful reward normalization
+- Always use `processing_class=` instead of deprecated `tokenizer=`
+
+**See:** `references/trl-fine-tuning.md` for full method references and `templates/` for starter scripts.
+
+## Section 3: Unsloth — Fast & Memory-Efficient Fine-Tuning
+
+2-5x faster training with 50-80% less memory. Optimized for LoRA/QLoRA on consumer GPUs.
+
+**When to use:** Single-GPU training, limited VRAM (4-24GB), rapid iteration, learning/experimentation.
+
+**Key features:**
+- 2-5x faster than standard HuggingFace training
+- 50-80% less VRAM usage
+- Auto-selects optimal LoRA/QLoRA parameters
+- Supports Llama, Mistral, Gemma, Qwen architectures
+- Drop-in replacement: same HuggingFace `Trainer` API
+
+**Quick start:**
+```python
+from unsloth import FastLanguageModel
+model, tokenizer = FastLanguageModel.from_pretrained(
+    model_name="unsloth/llama-3-8b-bnb-4bit",
+    max_seq_length=2048,
+    load_in_4bit=True,
+)
+model = FastLanguageModel.get_peft_model(
+    model, r=16,
+    lora_alpha=16, lora_dropout=0,
+    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
+)
+```
+
+**Pitfalls:**
+- Only supports specific architectures (Llama, Mistral, Gemma, Qwen)
+- May lag behind upstream Transformers version
+- Best for LoRA; full fine-tuning not the primary target
+
+**See:** `references/unsloth.md` for full documentation.
+
+## Comparison Table
+
+| Feature | Axolotl | TRL | Unsloth |
+|---------|---------|-----|---------|
+| SFT | ✅ | ✅ | ✅ |
+| DPO | ✅ | ✅ | ❌ |
+| PPO/GRPO | ✅ | ✅ | ❌ |
+| LoRA/QLoRA | ✅ | ✅ (via PEFT) | ✅ (native) |
+| Full fine-tune | ✅ | ✅ | ⚠️ Limited |
+| Multi-GPU | ✅ DeepSpeed | ✅ Accelerate | ⚠️ Limited |
+| YAML config | ✅ | ❌ Python | ❌ Python |
+| Speed | Baseline | Baseline | 2-5x faster |
+| Memory | Baseline | Baseline | 50-80% less |
+| Cloud backends | Modal, RunPod | Any | Any |
+
+## Workflow: Which Path?
+
+1. **Need the fastest iteration on limited hardware?** → Start with Unsloth
+2. **Need RLHF / preference alignment?** → Use TRL (or Axolotl's TRL backend)
+3. **Need production reproducibility + multi-method?** → Use Axolotl
+4. **Training complete?** → Export to GGUF (see `llm-inference` skill) or serve with vLLM