diff --git a/skills/mlops/llm-fine-tuning/SKILL.md b/skills/mlops/llm-fine-tuning/SKILL.md new file mode 100644 index 0000000..cf9679f --- /dev/null +++ b/skills/mlops/llm-fine-tuning/SKILL.md @@ -0,0 +1,173 @@ +--- +name: llm-fine-tuning +description: "LLM fine-tuning: framework comparison (Axolotl, TRL, Unsloth) and method selection (SFT, DPO, PPO, GRPO)." +version: 1.0.0 +author: Hermes Agent +license: MIT +metadata: + hermes: + tags: [fine-tuning, llm, axolotl, trl, unsloth, lora, qlora, dpo, ppo, grpo, rlhf, sft, training] + related_skills: [huggingface-hub, llm-inference] +--- + +# LLM Fine-Tuning + +Three frameworks for fine-tuning LLMs, each optimized for different use cases. Choose based on your training objective, then load the detailed reference for that framework. + +## Choosing a Framework + +| Need | Framework | Why | +|------|-----------|-----| +| Production-grade multi-method training | **Axolotl** | Most mature, YAML-driven, 100+ model presets, LoRA/QLoRA/DPO/KTO/ORPO/GRPO | +| Reinforcement learning from human feedback | **TRL** | Native SFT/DPO/PPO/GRPO, HuggingFace-native, research-friendly | +| Fastest training on limited hardware | **Unsloth** | 2-5x faster, 50-80% less memory, great for LoRA/QLoRA on single GPU | + +## Choosing a Method + +| Method | Use When | Training Data | +|--------|----------|---------------| +| **SFT** (Supervised Fine-Tuning) | Instruction tuning, domain adaptation | Prompt-completion pairs | +| **DPO** (Direct Preference Optimization) | Align with preferences, no reward model needed | Chosen/rejected pairs | +| **PPO** (Proximal Policy Optimization) | RLHF with online reward model | Prompt + reward model | +| **GRPO** (Group Relative Policy Optimization) | Rule-based rewards, no value model needed | Prompt + rule-based reward | +| **KTO/ORPO** | Simpler alignment, no pairwise preferences | Binary good/bad signals | +| **LoRA/QLoRA** | Parameter-efficient, limited GPU VRAM | (applies to any method above) | + +## Section 1: Axolotl — YAML-Driven Multi-Method Training + +Production-grade fine-tuning framework. YAML configs, 100+ model presets, LoRA/QLoRA, DPO/KTO/ORPO/GRPO, multimodal support. + +**When to use:** Production training jobs, multi-method experiments, need reproducibility, working with large models. + +**Key features:** +- YAML-driven configuration (no Python scripts needed) +- 100+ model presets (Llama, Mistral, Qwen, Gemma, etc.) +- DeepSpeed FSDP integration for multi-GPU +- LoRA/QLoRA/DPO/KTO/ORPO/GRPO support +- Multimodal fine-tuning (vision-language models) +- Cloud backends (Modal, RunPod) + +**Quick start:** +```bash +pip install axolotl +# Create a YAML config, then: +accelerate launch -m axolotl.cli.train config.yaml +``` + +**Key config pattern (FSDP v2):** +```yaml +fsdp_version: 2 +fsdp_config: + offload_params: true + state_dict_type: FULL_STATE_DICT + auto_wrap_policy: TRANSFORMER_BASED_WRAP +``` + +**Pitfalls:** +- `context_parallel_size` must divide total GPUs +- `save_compressed: true` reduces disk 40%, compatible with vLLM +- Integration code need not go in `integrations/` folder +- Handle both single-example and batched data shapes in custom collators + +**See:** `references/axolotl.md` for full config reference, API docs, and dataset formats. + +## Section 2: TRL — Transformer Reinforcement Learning + +HuggingFace-native post-training library. SFT, DPO, PPO, GRPO, and reward model training. + +**When to use:** RLHF workflows, preference alignment, research experiments, need tight HuggingFace integration. + +**Key features:** +- `SFTTrainer` — supervised instruction tuning +- `DPOTrainer` — direct preference optimization +- `PPOTrainer` — proximal policy optimization (classic RLHF) +- `GRPOTrainer` — group relative policy optimization (no value model) +- Reward model training and online RL methods +- Works with any HuggingFace model + PEFT for LoRA + +**Quick start (SFT):** +```python +from trl import SFTTrainer +trainer = SFTTrainer( + model="Qwen/Qwen2.5-0.5B", + train_dataset=dataset, +) +trainer.train() +``` + +**Quick start (DPO):** +```python +from trl import DPOTrainer, DPOConfig +config = DPOConfig(output_dir="model-dpo", beta=0.1) +trainer = DPOTrainer( + model=model, args=config, + train_dataset=preference_dataset, + processing_class=tokenizer +) +trainer.train() +``` + +**Pitfalls:** +- DPO `beta` controls alignment strength — too high reduces diversity +- PPO requires a reward model + reference model (memory-intensive) +- GRPO eliminates the value model but needs careful reward normalization +- Always use `processing_class=` instead of deprecated `tokenizer=` + +**See:** `references/trl-fine-tuning.md` for full method references and `templates/` for starter scripts. + +## Section 3: Unsloth — Fast & Memory-Efficient Fine-Tuning + +2-5x faster training with 50-80% less memory. Optimized for LoRA/QLoRA on consumer GPUs. + +**When to use:** Single-GPU training, limited VRAM (4-24GB), rapid iteration, learning/experimentation. + +**Key features:** +- 2-5x faster than standard HuggingFace training +- 50-80% less VRAM usage +- Auto-selects optimal LoRA/QLoRA parameters +- Supports Llama, Mistral, Gemma, Qwen architectures +- Drop-in replacement: same HuggingFace `Trainer` API + +**Quick start:** +```python +from unsloth import FastLanguageModel +model, tokenizer = FastLanguageModel.from_pretrained( + model_name="unsloth/llama-3-8b-bnb-4bit", + max_seq_length=2048, + load_in_4bit=True, +) +model = FastLanguageModel.get_peft_model( + model, r=16, + lora_alpha=16, lora_dropout=0, + target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], +) +``` + +**Pitfalls:** +- Only supports specific architectures (Llama, Mistral, Gemma, Qwen) +- May lag behind upstream Transformers version +- Best for LoRA; full fine-tuning not the primary target + +**See:** `references/unsloth.md` for full documentation. + +## Comparison Table + +| Feature | Axolotl | TRL | Unsloth | +|---------|---------|-----|---------| +| SFT | ✅ | ✅ | ✅ | +| DPO | ✅ | ✅ | ❌ | +| PPO/GRPO | ✅ | ✅ | ❌ | +| LoRA/QLoRA | ✅ | ✅ (via PEFT) | ✅ (native) | +| Full fine-tune | ✅ | ✅ | ⚠️ Limited | +| Multi-GPU | ✅ DeepSpeed | ✅ Accelerate | ⚠️ Limited | +| YAML config | ✅ | ❌ Python | ❌ Python | +| Speed | Baseline | Baseline | 2-5x faster | +| Memory | Baseline | Baseline | 50-80% less | +| Cloud backends | Modal, RunPod | Any | Any | + +## Workflow: Which Path? + +1. **Need the fastest iteration on limited hardware?** → Start with Unsloth +2. **Need RLHF / preference alignment?** → Use TRL (or Axolotl's TRL backend) +3. **Need production reproducibility + multi-method?** → Use Axolotl +4. **Training complete?** → Export to GGUF (see `llm-inference` skill) or serve with vLLM