atomk-hermes-skills/skills/mlops/computer-vision-audio-models/SKILL.md

name, description, version, author, license, metadata

name	description	version	author	license	metadata
computer-vision-audio-models	Specialized AI models: image segmentation (SAM / Segment Anything) and audio generation (AudioCraft / MusicGen).	1.0.0	Hermes Agent	MIT	hermes tags related_skills computer-vision audio-generation sam segment-anything audiocraft musicgen audiogen image-segmentation comfyui llm-inference

Computer Vision & Audio Models

Two specialized model families for non-text AI tasks: zero-shot image segmentation and text-to-audio generation.

Section 1: Segment Anything Model (SAM) — Zero-Shot Image Segmentation

Meta's Segment Anything Model for zero-shot image segmentation via points, boxes, or masks as prompts.

When to use: Object segmentation, image editing, data annotation, medical imaging, removing/replacing objects in images.

Key features:

• Zero-shot segmentation — no training data needed for new objects
• Point, box, and mask prompt types
• SAM 2 supports video segmentation with memory
• Automatic mask generation (segment everything)
• ONNX export for production deployment

Quick start:

from segment_anything import SamPredictor, sam_model_registry
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)
predictor.set_image(image)
masks, scores, logits = predictor.predict(point_coords=input_point, point_labels=input_label)

Pitfalls:

• ViT-H checkpoint is ~2.4GB; use ViT-B (~375MB) for limited VRAM
• Point prompts need point_labels (1=foreground, 0=background)
• For SAM 2 video segmentation, frames must be processed sequentially
• Automatic mask generation is slow on CPU — use GPU

See: references/segment-anything.md for full API, advanced usage, and troubleshooting.

Section 2: AudioCraft — Text-to-Audio Generation

Meta's AudioCraft for MusicGen (text-to-music) and AudioGen (text-to-sound effects).

When to use: Generating music from text descriptions, creating sound effects, audio for games/video, ambient soundscapes.

Key features:

• MusicGen: text-to-music with melody conditioning
• AudioGen: text-to-sound-effects
• Multiple model sizes (small/medium/large/melody)
• Streaming generation for real-time preview
• Fine-tuning support on custom audio data

Quick start (MusicGen):

from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained('medium')
model.set_generation_params(duration=8)
wav = model.generate(['happy rock song with guitar solo'])
audio_write('output', wav[0].cpu(), model.sample_rate, strategy="loudness")

Quick start (AudioGen):

from audiocraft.models import AudioGen
model = AudioGen.get_pretrained('medium')
model.set_generation_params(duration=5)
wav = model.generate(['dog barking in the distance'])

Pitfalls:

• Large model requires ~16GB VRAM for generation; medium works on ~8GB
• loudness normalization strategy avoids clipping
• Melody conditioning requires a reference audio file (24kHz mono)
• Fine-tuning needs hours of clean audio data + significant GPU time

See: references/audiocraft.md for full API, advanced usage, and troubleshooting.