Add mlops/llm-evaluation

skills/mlops/llm-evaluation/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: llm-evaluation
+description: "LLM evaluation: benchmarking (lm-eval-harness) and experiment tracking (Weights & Biases)."
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+metadata:
+  hermes:
+    tags: [evaluation, llm, benchmarks, mmlu, gsm8k, wandb, experiment-tracking, lm-eval]
+    related_skills: [llm-fine-tuning, huggingface-hub]
+---
+
+# LLM Evaluation
+
+Two complementary evaluation workflows: standardized benchmarking with lm-eval-harness, and experiment tracking with Weights & Biases.
+
+## Section 1: lm-evaluation-harness — Standardized Benchmarking
+
+Run standardized LLM benchmarks (MMLU, GSM8K, HellaSwag, etc.) using EleutherAI's lm-eval-harness.
+
+**When to use:** Comparing model performance, evaluating fine-tuning results, academic benchmarking, model selection.
+
+**Key features:**
+- 60+ standard benchmarks (MMLU, GSM8K, HellaSwag, Arc, TruthfulQA, WinoGrande, etc.)
+- Supports HuggingFace, vLLM, OpenAI, and local GGUF models
+- Few-shot evaluation with configurable shot counts
+- Custom task creation via YAML
+- Distributed evaluation with Ray
+
+**Quick start:**
+```bash
+pip install lm-eval
+lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B --tasks mmlu,gsm8k
+```
+
+**Custom task example:**
+```yaml
+task: my_custom_task
+dataset_path: json
+dataset_kwargs:
+  data_files:
+    test: my_data.jsonl
+output_type: multiple_choice
+doc_to_text: "{{question}}"
+doc_to_target: "{{answer}}"
+doc_to_choice: "{{choices}}"
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+```
+
+**See:** `references/lm-evaluation-harness.md` for full API, custom tasks, and distributed eval guides.
+
+## Section 2: Weights & Biases — Experiment Tracking
+
+Track ML experiments, visualize metrics, manage sweeps, and collaborate on model development.
+
+**When to use:** Tracking training runs, comparing experiments, hyperparameter sweeps, team collaboration, model registry.
+
+**Key features:**
+- Automatic metric logging from HuggingFace Trainer, Axolotl, TRL
+- Hyperparameter sweeps (grid, random, Bayesian)
+- Artifact versioning (datasets, models)
+- Model registry for staging → production
+- Team dashboards and collaboration
+
+**Quick start:**
+```bash
+pip install wandb
+wandb login
+```
+
+```python
+import wandb
+wandb.init(project="my-llm-project", config={"lr": 2e-5, "epochs": 3})
+# Automatic with HuggingFace Trainer:
+# trainer = Trainer(args=TrainingArguments(report_to="wandb"), ...)
+```
+
+**See:** `references/weights-and-biases.md` for integrations, sweeps, and artifacts guides.
+
+## Evaluation Workflow
+
+1. **During training:** Log metrics to W&B (loss, eval accuracy, generation samples)
+2. **After training:** Run lm-eval-harness on the checkpoint
+3. **Comparison:** Log lm-eval results to W&B for visual comparison across runs
+4. **Model selection:** Use W&B model registry to promote best checkpoint to production