atomk-hermes-skills/skills/mlops/llm-evaluation/SKILL.md

name, description, version, author, license, metadata

name	description	version	author	license	metadata
llm-evaluation	LLM evaluation: benchmarking (lm-eval-harness) and experiment tracking (Weights & Biases).	1.0.0	Hermes Agent	MIT	hermes tags related_skills evaluation llm benchmarks mmlu gsm8k wandb experiment-tracking lm-eval llm-fine-tuning huggingface-hub

LLM Evaluation

Two complementary evaluation workflows: standardized benchmarking with lm-eval-harness, and experiment tracking with Weights & Biases.

Section 1: lm-evaluation-harness — Standardized Benchmarking

Run standardized LLM benchmarks (MMLU, GSM8K, HellaSwag, etc.) using EleutherAI's lm-eval-harness.

When to use: Comparing model performance, evaluating fine-tuning results, academic benchmarking, model selection.

Key features:

• 60+ standard benchmarks (MMLU, GSM8K, HellaSwag, Arc, TruthfulQA, WinoGrande, etc.)
• Supports HuggingFace, vLLM, OpenAI, and local GGUF models
• Few-shot evaluation with configurable shot counts
• Custom task creation via YAML
• Distributed evaluation with Ray

Quick start:

pip install lm-eval
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B --tasks mmlu,gsm8k

Custom task example:

task: my_custom_task
dataset_path: json
dataset_kwargs:
  data_files:
    test: my_data.jsonl
output_type: multiple_choice
doc_to_text: "{{question}}"
doc_to_target: "{{answer}}"
doc_to_choice: "{{choices}}"
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true

See: references/lm-evaluation-harness.md for full API, custom tasks, and distributed eval guides.

Section 2: Weights & Biases — Experiment Tracking

Track ML experiments, visualize metrics, manage sweeps, and collaborate on model development.

When to use: Tracking training runs, comparing experiments, hyperparameter sweeps, team collaboration, model registry.

Key features:

• Automatic metric logging from HuggingFace Trainer, Axolotl, TRL
• Hyperparameter sweeps (grid, random, Bayesian)
• Artifact versioning (datasets, models)
• Model registry for staging → production
• Team dashboards and collaboration

Quick start:

pip install wandb
wandb login

import wandb
wandb.init(project="my-llm-project", config={"lr": 2e-5, "epochs": 3})
# Automatic with HuggingFace Trainer:
# trainer = Trainer(args=TrainingArguments(report_to="wandb"), ...)

See: references/weights-and-biases.md for integrations, sweeps, and artifacts guides.

Evaluation Workflow

1. During training: Log metrics to W&B (loss, eval accuracy, generation samples)
2. After training: Run lm-eval-harness on the checkpoint
3. Comparison: Log lm-eval results to W&B for visual comparison across runs
4. Model selection: Use W&B model registry to promote best checkpoint to production