89 lines
2.8 KiB
Markdown
89 lines
2.8 KiB
Markdown
---
|
|
name: llm-evaluation
|
|
description: "LLM evaluation: benchmarking (lm-eval-harness) and experiment tracking (Weights & Biases)."
|
|
version: 1.0.0
|
|
author: Hermes Agent
|
|
license: MIT
|
|
metadata:
|
|
hermes:
|
|
tags: [evaluation, llm, benchmarks, mmlu, gsm8k, wandb, experiment-tracking, lm-eval]
|
|
related_skills: [llm-fine-tuning, huggingface-hub]
|
|
---
|
|
|
|
# LLM Evaluation
|
|
|
|
Two complementary evaluation workflows: standardized benchmarking with lm-eval-harness, and experiment tracking with Weights & Biases.
|
|
|
|
## Section 1: lm-evaluation-harness — Standardized Benchmarking
|
|
|
|
Run standardized LLM benchmarks (MMLU, GSM8K, HellaSwag, etc.) using EleutherAI's lm-eval-harness.
|
|
|
|
**When to use:** Comparing model performance, evaluating fine-tuning results, academic benchmarking, model selection.
|
|
|
|
**Key features:**
|
|
- 60+ standard benchmarks (MMLU, GSM8K, HellaSwag, Arc, TruthfulQA, WinoGrande, etc.)
|
|
- Supports HuggingFace, vLLM, OpenAI, and local GGUF models
|
|
- Few-shot evaluation with configurable shot counts
|
|
- Custom task creation via YAML
|
|
- Distributed evaluation with Ray
|
|
|
|
**Quick start:**
|
|
```bash
|
|
pip install lm-eval
|
|
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B --tasks mmlu,gsm8k
|
|
```
|
|
|
|
**Custom task example:**
|
|
```yaml
|
|
task: my_custom_task
|
|
dataset_path: json
|
|
dataset_kwargs:
|
|
data_files:
|
|
test: my_data.jsonl
|
|
output_type: multiple_choice
|
|
doc_to_text: "{{question}}"
|
|
doc_to_target: "{{answer}}"
|
|
doc_to_choice: "{{choices}}"
|
|
metric_list:
|
|
- metric: acc
|
|
aggregation: mean
|
|
higher_is_better: true
|
|
```
|
|
|
|
**See:** `references/lm-evaluation-harness.md` for full API, custom tasks, and distributed eval guides.
|
|
|
|
## Section 2: Weights & Biases — Experiment Tracking
|
|
|
|
Track ML experiments, visualize metrics, manage sweeps, and collaborate on model development.
|
|
|
|
**When to use:** Tracking training runs, comparing experiments, hyperparameter sweeps, team collaboration, model registry.
|
|
|
|
**Key features:**
|
|
- Automatic metric logging from HuggingFace Trainer, Axolotl, TRL
|
|
- Hyperparameter sweeps (grid, random, Bayesian)
|
|
- Artifact versioning (datasets, models)
|
|
- Model registry for staging → production
|
|
- Team dashboards and collaboration
|
|
|
|
**Quick start:**
|
|
```bash
|
|
pip install wandb
|
|
wandb login
|
|
```
|
|
|
|
```python
|
|
import wandb
|
|
wandb.init(project="my-llm-project", config={"lr": 2e-5, "epochs": 3})
|
|
# Automatic with HuggingFace Trainer:
|
|
# trainer = Trainer(args=TrainingArguments(report_to="wandb"), ...)
|
|
```
|
|
|
|
**See:** `references/weights-and-biases.md` for integrations, sweeps, and artifacts guides.
|
|
|
|
## Evaluation Workflow
|
|
|
|
1. **During training:** Log metrics to W&B (loss, eval accuracy, generation samples)
|
|
2. **After training:** Run lm-eval-harness on the checkpoint
|
|
3. **Comparison:** Log lm-eval results to W&B for visual comparison across runs
|
|
4. **Model selection:** Use W&B model registry to promote best checkpoint to production
|