---
name: computer-vision-audio-models
description: "Specialized AI models: image segmentation (SAM / Segment Anything) and audio generation (AudioCraft / MusicGen)."
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [computer-vision, audio-generation, sam, segment-anything, audiocraft, musicgen, audiogen, image-segmentation]
    related_skills: [comfyui, llm-inference]
---

# Computer Vision & Audio Models

Two specialized model families for non-text AI tasks: zero-shot image segmentation and text-to-audio generation.

## Section 1: Segment Anything Model (SAM) — Zero-Shot Image Segmentation

Meta's Segment Anything Model for zero-shot image segmentation via points, boxes, or masks as prompts.

**When to use:** Object segmentation, image editing, data annotation, medical imaging, removing/replacing objects in images.

**Key features:**
- Zero-shot segmentation — no training data needed for new objects
- Point, box, and mask prompt types
- SAM 2 supports video segmentation with memory
- Automatic mask generation (segment everything)
- ONNX export for production deployment

**Quick start:**
```python
from segment_anything import SamPredictor, sam_model_registry
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)
predictor.set_image(image)
masks, scores, logits = predictor.predict(point_coords=input_point, point_labels=input_label)
```

**Pitfalls:**
- ViT-H checkpoint is ~2.4GB; use ViT-B (~375MB) for limited VRAM
- Point prompts need `point_labels` (1=foreground, 0=background)
- For SAM 2 video segmentation, frames must be processed sequentially
- Automatic mask generation is slow on CPU — use GPU

**See:** `references/segment-anything.md` for full API, advanced usage, and troubleshooting.

## Section 2: AudioCraft — Text-to-Audio Generation

Meta's AudioCraft for MusicGen (text-to-music) and AudioGen (text-to-sound effects).

**When to use:** Generating music from text descriptions, creating sound effects, audio for games/video, ambient soundscapes.

**Key features:**
- MusicGen: text-to-music with melody conditioning
- AudioGen: text-to-sound-effects
- Multiple model sizes (small/medium/large/melody)
- Streaming generation for real-time preview
- Fine-tuning support on custom audio data

**Quick start (MusicGen):**
```python
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained('medium')
model.set_generation_params(duration=8)
wav = model.generate(['happy rock song with guitar solo'])
audio_write('output', wav[0].cpu(), model.sample_rate, strategy="loudness")
```

**Quick start (AudioGen):**
```python
from audiocraft.models import AudioGen
model = AudioGen.get_pretrained('medium')
model.set_generation_params(duration=5)
wav = model.generate(['dog barking in the distance'])
```

**Pitfalls:**
- Large model requires ~16GB VRAM for generation; medium works on ~8GB
- `loudness` normalization strategy avoids clipping
- Melody conditioning requires a reference audio file (24kHz mono)
- Fine-tuning needs hours of clean audio data + significant GPU time

**See:** `references/audiocraft.md` for full API, advanced usage, and troubleshooting.