diff --git a/skills/mlops/computer-vision-audio-models/SKILL.md b/skills/mlops/computer-vision-audio-models/SKILL.md new file mode 100644 index 0000000..5b8419c --- /dev/null +++ b/skills/mlops/computer-vision-audio-models/SKILL.md @@ -0,0 +1,85 @@ +--- +name: computer-vision-audio-models +description: "Specialized AI models: image segmentation (SAM / Segment Anything) and audio generation (AudioCraft / MusicGen)." +version: 1.0.0 +author: Hermes Agent +license: MIT +metadata: + hermes: + tags: [computer-vision, audio-generation, sam, segment-anything, audiocraft, musicgen, audiogen, image-segmentation] + related_skills: [comfyui, llm-inference] +--- + +# Computer Vision & Audio Models + +Two specialized model families for non-text AI tasks: zero-shot image segmentation and text-to-audio generation. + +## Section 1: Segment Anything Model (SAM) — Zero-Shot Image Segmentation + +Meta's Segment Anything Model for zero-shot image segmentation via points, boxes, or masks as prompts. + +**When to use:** Object segmentation, image editing, data annotation, medical imaging, removing/replacing objects in images. + +**Key features:** +- Zero-shot segmentation — no training data needed for new objects +- Point, box, and mask prompt types +- SAM 2 supports video segmentation with memory +- Automatic mask generation (segment everything) +- ONNX export for production deployment + +**Quick start:** +```python +from segment_anything import SamPredictor, sam_model_registry +sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth") +predictor = SamPredictor(sam) +predictor.set_image(image) +masks, scores, logits = predictor.predict(point_coords=input_point, point_labels=input_label) +``` + +**Pitfalls:** +- ViT-H checkpoint is ~2.4GB; use ViT-B (~375MB) for limited VRAM +- Point prompts need `point_labels` (1=foreground, 0=background) +- For SAM 2 video segmentation, frames must be processed sequentially +- Automatic mask generation is slow on CPU — use GPU + +**See:** `references/segment-anything.md` for full API, advanced usage, and troubleshooting. + +## Section 2: AudioCraft — Text-to-Audio Generation + +Meta's AudioCraft for MusicGen (text-to-music) and AudioGen (text-to-sound effects). + +**When to use:** Generating music from text descriptions, creating sound effects, audio for games/video, ambient soundscapes. + +**Key features:** +- MusicGen: text-to-music with melody conditioning +- AudioGen: text-to-sound-effects +- Multiple model sizes (small/medium/large/melody) +- Streaming generation for real-time preview +- Fine-tuning support on custom audio data + +**Quick start (MusicGen):** +```python +from audiocraft.models import MusicGen +from audiocraft.data.audio import audio_write + +model = MusicGen.get_pretrained('medium') +model.set_generation_params(duration=8) +wav = model.generate(['happy rock song with guitar solo']) +audio_write('output', wav[0].cpu(), model.sample_rate, strategy="loudness") +``` + +**Quick start (AudioGen):** +```python +from audiocraft.models import AudioGen +model = AudioGen.get_pretrained('medium') +model.set_generation_params(duration=5) +wav = model.generate(['dog barking in the distance']) +``` + +**Pitfalls:** +- Large model requires ~16GB VRAM for generation; medium works on ~8GB +- `loudness` normalization strategy avoids clipping +- Melody conditioning requires a reference audio file (24kHz mono) +- Fine-tuning needs hours of clean audio data + significant GPU time + +**See:** `references/audiocraft.md` for full API, advanced usage, and troubleshooting.