diff --git a/skills/web/wechat-article-extraction/SKILL.md b/skills/web/wechat-article-extraction/SKILL.md new file mode 100644 index 0000000..39d2415 --- /dev/null +++ b/skills/web/wechat-article-extraction/SKILL.md @@ -0,0 +1,108 @@ +--- +name: wechat-article-extraction +description: Extract full article content from WeChat public account (微信公众号) articles at mp.weixin.qq.com. Plain requests with desktop UA works; all "smarter" tools fail. +version: 1.0 +tags: [wechat, scraping, article, chinese] +--- + +# WeChat Article Extraction (mp.weixin.qq.com) + +Extract full article content from WeChat public account (微信公众号) articles. + +## What Works + +**Plain `requests` with desktop Chrome User-Agent.** The full article HTML is server-rendered and included in the initial response — no JS rendering needed. + +```python +import requests, re, html + +session = requests.Session() +headers = { + 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36', + 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', + 'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8', +} +resp = session.get(url, headers=headers, timeout=20) +content = resp.text + +# Title +title = re.search(r'var\s+msg_title\s*=\s*["\'](.+?)["\']', content) +title_text = html.unescape(title.group(1)) if title else None + +# Article body +body = re.search(r'id="js_content"[^>]*>(.*?)', content, re.DOTALL) +text = re.sub(r'<[^>]+>', '\n', body.group(1)) +text = html.unescape(text) +text = re.sub(r'\n{3,}', '\n\n', text).strip() +``` + +## What Does NOT Work + +| Tool | Result | Reason | +|------|--------|--------| +| Jina Reader (`r.jina.ai`) | ❌ CAPTCHA block | Detected as bot, returns "环境异常" | +| Crawl4AI (Playwright) | ❌ Timeout | Anti-bot blocks `js_content` selector from appearing | +| Browser automation | ❌ Sandbox/CAPTCHA | Chrome sandbox issues + verification wall | +| Google Cache | ❌ No cache | Google hasn't cached most WeChat articles | +| Mobile UA (MicroMessenger) | ❌ Empty content | Triggers anti-bot, content hidden behind JS | +| SerpAPI search | ❌ No results | WeChat articles rarely indexed | + +## Extraction Strategy (Two Patterns) + +WeChat articles come in two rendering variants. Try **Pattern 1** first, fall back to **Pattern 2**. + +### Pattern 1: Server-rendered HTML (`js_content` div) + +Most common. The full article HTML is in `
]*>', '\n', text) + text = re.sub(r'
', '', text) + text = re.sub(r'<[^>]+>', '', text) + text = html.unescape(text) +``` + +### Pattern 2: JS-string embedded (`window.msg_title`) + +Some shorter articles embed the ENTIRE content inside a JS string assignment. +The `