Add mlops/computer-vision-audio-models

skills/mlops/computer-vision-audio-models/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: computer-vision-audio-models
+description: "Specialized AI models: image segmentation (SAM / Segment Anything) and audio generation (AudioCraft / MusicGen)."
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+metadata:
+  hermes:
+    tags: [computer-vision, audio-generation, sam, segment-anything, audiocraft, musicgen, audiogen, image-segmentation]
+    related_skills: [comfyui, llm-inference]
+---
+
+# Computer Vision & Audio Models
+
+Two specialized model families for non-text AI tasks: zero-shot image segmentation and text-to-audio generation.
+
+## Section 1: Segment Anything Model (SAM) — Zero-Shot Image Segmentation
+
+Meta's Segment Anything Model for zero-shot image segmentation via points, boxes, or masks as prompts.
+
+**When to use:** Object segmentation, image editing, data annotation, medical imaging, removing/replacing objects in images.
+
+**Key features:**
+- Zero-shot segmentation — no training data needed for new objects
+- Point, box, and mask prompt types
+- SAM 2 supports video segmentation with memory
+- Automatic mask generation (segment everything)
+- ONNX export for production deployment
+
+**Quick start:**
+```python
+from segment_anything import SamPredictor, sam_model_registry
+sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
+predictor = SamPredictor(sam)
+predictor.set_image(image)
+masks, scores, logits = predictor.predict(point_coords=input_point, point_labels=input_label)
+```
+
+**Pitfalls:**
+- ViT-H checkpoint is ~2.4GB; use ViT-B (~375MB) for limited VRAM
+- Point prompts need `point_labels` (1=foreground, 0=background)
+- For SAM 2 video segmentation, frames must be processed sequentially
+- Automatic mask generation is slow on CPU — use GPU
+
+**See:** `references/segment-anything.md` for full API, advanced usage, and troubleshooting.
+
+## Section 2: AudioCraft — Text-to-Audio Generation
+
+Meta's AudioCraft for MusicGen (text-to-music) and AudioGen (text-to-sound effects).
+
+**When to use:** Generating music from text descriptions, creating sound effects, audio for games/video, ambient soundscapes.
+
+**Key features:**
+- MusicGen: text-to-music with melody conditioning
+- AudioGen: text-to-sound-effects
+- Multiple model sizes (small/medium/large/melody)
+- Streaming generation for real-time preview
+- Fine-tuning support on custom audio data
+
+**Quick start (MusicGen):**
+```python
+from audiocraft.models import MusicGen
+from audiocraft.data.audio import audio_write
+
+model = MusicGen.get_pretrained('medium')
+model.set_generation_params(duration=8)
+wav = model.generate(['happy rock song with guitar solo'])
+audio_write('output', wav[0].cpu(), model.sample_rate, strategy="loudness")
+```
+
+**Quick start (AudioGen):**
+```python
+from audiocraft.models import AudioGen
+model = AudioGen.get_pretrained('medium')
+model.set_generation_params(duration=5)
+wav = model.generate(['dog barking in the distance'])
+```
+
+**Pitfalls:**
+- Large model requires ~16GB VRAM for generation; medium works on ~8GB
+- `loudness` normalization strategy avoids clipping
+- Melody conditioning requires a reference audio file (24kHz mono)
+- Fine-tuning needs hours of clean audio data + significant GPU time
+
+**See:** `references/audiocraft.md` for full API, advanced usage, and troubleshooting.