---
name: llm-evaluation
description: "LLM evaluation: benchmarking (lm-eval-harness) and experiment tracking (Weights & Biases)."
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [evaluation, llm, benchmarks, mmlu, gsm8k, wandb, experiment-tracking, lm-eval]
    related_skills: [llm-fine-tuning, huggingface-hub]
---

# LLM Evaluation

Two complementary evaluation workflows: standardized benchmarking with lm-eval-harness, and experiment tracking with Weights & Biases.

## Section 1: lm-evaluation-harness — Standardized Benchmarking

Run standardized LLM benchmarks (MMLU, GSM8K, HellaSwag, etc.) using EleutherAI's lm-eval-harness.

**When to use:** Comparing model performance, evaluating fine-tuning results, academic benchmarking, model selection.

**Key features:**
- 60+ standard benchmarks (MMLU, GSM8K, HellaSwag, Arc, TruthfulQA, WinoGrande, etc.)
- Supports HuggingFace, vLLM, OpenAI, and local GGUF models
- Few-shot evaluation with configurable shot counts
- Custom task creation via YAML
- Distributed evaluation with Ray

**Quick start:**
```bash
pip install lm-eval
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B --tasks mmlu,gsm8k
```

**Custom task example:**
```yaml
task: my_custom_task
dataset_path: json
dataset_kwargs:
  data_files:
    test: my_data.jsonl
output_type: multiple_choice
doc_to_text: "{{question}}"
doc_to_target: "{{answer}}"
doc_to_choice: "{{choices}}"
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
```

**See:** `references/lm-evaluation-harness.md` for full API, custom tasks, and distributed eval guides.

## Section 2: Weights & Biases — Experiment Tracking

Track ML experiments, visualize metrics, manage sweeps, and collaborate on model development.

**When to use:** Tracking training runs, comparing experiments, hyperparameter sweeps, team collaboration, model registry.

**Key features:**
- Automatic metric logging from HuggingFace Trainer, Axolotl, TRL
- Hyperparameter sweeps (grid, random, Bayesian)
- Artifact versioning (datasets, models)
- Model registry for staging → production
- Team dashboards and collaboration

**Quick start:**
```bash
pip install wandb
wandb login
```

```python
import wandb
wandb.init(project="my-llm-project", config={"lr": 2e-5, "epochs": 3})
# Automatic with HuggingFace Trainer:
# trainer = Trainer(args=TrainingArguments(report_to="wandb"), ...)
```

**See:** `references/weights-and-biases.md` for integrations, sweeps, and artifacts guides.

## Evaluation Workflow

1. **During training:** Log metrics to W&B (loss, eval accuracy, generation samples)
2. **After training:** Run lm-eval-harness on the checkpoint
3. **Comparison:** Log lm-eval results to W&B for visual comparison across runs
4. **Model selection:** Use W&B model registry to promote best checkpoint to production