WeChat Article Extraction (mp.weixin.qq.com)

Extract full article content from WeChat public account (微信公众号) articles.

What Works

Plain requests with desktop Chrome User-Agent. The full article HTML is server-rendered and included in the initial response — no JS rendering needed.

import requests, re, html

session = requests.Session()
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
}
resp = session.get(url, headers=headers, timeout=20)
content = resp.text

# Title
title = re.search(r'var\s+msg_title\s*=\s*["\'](.+?)["\']', content)
title_text = html.unescape(title.group(1)) if title else None

# Article body
body = re.search(r'id="js_content"[^>]*>(.*?)</div>', content, re.DOTALL)
text = re.sub(r'<[^>]+>', '\n', body.group(1))
text = html.unescape(text)
text = re.sub(r'\n{3,}', '\n\n', text).strip()

What Does NOT Work

Tool	Result	Reason
Jina Reader (`r.jina.ai`)	❌ CAPTCHA block	Detected as bot, returns "环境异常"
Crawl4AI (Playwright)	❌ Timeout	Anti-bot blocks `js_content` selector from appearing
Browser automation	❌ Sandbox/CAPTCHA	Chrome sandbox issues + verification wall
Google Cache	❌ No cache	Google hasn't cached most WeChat articles
Mobile UA (MicroMessenger)	❌ Empty content	Triggers anti-bot, content hidden behind JS
SerpAPI search	❌ No results	WeChat articles rarely indexed

Extraction Strategy (Two Patterns)

WeChat articles come in two rendering variants. Try Pattern 1 first, fall back to Pattern 2.

Pattern 1: Server-rendered HTML (`js_content` div)

Most common. The full article HTML is in <div id="js_content">...</div>.

body = re.search(r'id="js_content"[^>]*>(.*?)</div>', content, re.DOTALL)
if body and body.group(1).strip():
    text = re.sub(r'<br\s*/?>', '\n', body.group(1))
    text = re.sub(r'<p[^>]*>', '\n', text)
    text = re.sub(r'</p>', '', text)
    text = re.sub(r'<[^>]+>', '', text)
    text = html.unescape(text)

Pattern 2: JS-string embedded (`window.msg_title`)

Some shorter articles embed the ENTIRE content inside a JS string assignment. The <div id="js_content"> is empty — content is populated dynamically by JS. The article text lives in window.msg_title = '...' with \\n for newlines and \\' for escaped quotes.

# Try Pattern 2 when js_content is empty
title_match = re.search(r"window\.msg_title\s*=\s*window\.title\s*=\s*'(.+?)'\s*\|\|\s*", content, re.S)
if not title_match:
    title_match = re.search(r"window\.msg_title\s*=\s*window\.title\s*=\s*'(.+?)';(?:\s*$|\s*\n)", content, re.S)
if title_match:
    article = title_match.group(1).replace('\\n', '\n').replace("\\'", "'").replace('\\\\', '\\')
    # First line is the title, rest is body
    lines = article.strip().split('\n')
    title = lines[0]
    body = '\n'.join(lines[1:])

The msg_title variable is also the source of the <title> tag — it's a single JS string that contains the full article text. The regex MUST use ' (single quotes) because WeChat uses single-quoted JS strings for this value.

Key Patterns

Title: var msg_title = "..."; — HTML-escaped, use html.unescape()
Description: var msg_desc = "...";
Body (Pattern 1): <div id="js_content">...</div> — server-rendered HTML with inline tags
Body (Pattern 2): window.msg_title = '...\\n...'; — JS-embedded, \\n for newlines
Images: data-src attribute (not src) — WeChat lazy-loads images
Author: var nickname = "..."; in JS

Pitfalls

Desktop UA is critical — mobile/bot user agents trigger the verification wall even with requests
Pattern 2: js_content div exists but is empty — don't assume failure; check window.msg_title as fallback
Pattern 2: use single quotes in regex — WeChat uses '...' not "..." for msg_title JS strings
Pattern 2: \\n not \n — the JS string literal has escaped newlines; use .replace('\\n', '\n')
Image URLs need transformation — data-src values are relative (//mmbiz.qpic.cn/...), prepend https:
Don't over-engineer — sophisticated tools (Crawl4AI, CamouFox) are overkill and actually worse for WeChat articles

4.7 KiB Raw Blame History