---
name: wechat-article-extraction
description: Extract full article content from WeChat public account (微信公众号) articles at mp.weixin.qq.com. Plain requests with desktop UA works; all "smarter" tools fail.
version: 1.0
tags: [wechat, scraping, article, chinese]
---

# WeChat Article Extraction (mp.weixin.qq.com)

Extract full article content from WeChat public account (微信公众号) articles.

## What Works

**Plain `requests` with desktop Chrome User-Agent.** The full article HTML is server-rendered and included in the initial response — no JS rendering needed.

```python
import requests, re, html

session = requests.Session()
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
}
resp = session.get(url, headers=headers, timeout=20)
content = resp.text

# Title
title = re.search(r'var\s+msg_title\s*=\s*["\'](.+?)["\']', content)
title_text = html.unescape(title.group(1)) if title else None

# Article body
body = re.search(r'id="js_content"[^>]*>(.*?)</div>', content, re.DOTALL)
text = re.sub(r'<[^>]+>', '\n', body.group(1))
text = html.unescape(text)
text = re.sub(r'\n{3,}', '\n\n', text).strip()
```

## What Does NOT Work

| Tool | Result | Reason |
|------|--------|--------|
| Jina Reader (`r.jina.ai`) | ❌ CAPTCHA block | Detected as bot, returns "环境异常" |
| Crawl4AI (Playwright) | ❌ Timeout | Anti-bot blocks `js_content` selector from appearing |
| Browser automation | ❌ Sandbox/CAPTCHA | Chrome sandbox issues + verification wall |
| Google Cache | ❌ No cache | Google hasn't cached most WeChat articles |
| Mobile UA (MicroMessenger) | ❌ Empty content | Triggers anti-bot, content hidden behind JS |
| SerpAPI search | ❌ No results | WeChat articles rarely indexed |

## Extraction Strategy (Two Patterns)

WeChat articles come in two rendering variants. Try **Pattern 1** first, fall back to **Pattern 2**.

### Pattern 1: Server-rendered HTML (`js_content` div)

Most common. The full article HTML is in `<div id="js_content">...</div>`.

```python
body = re.search(r'id="js_content"[^>]*>(.*?)</div>', content, re.DOTALL)
if body and body.group(1).strip():
    text = re.sub(r'<br\s*/?>', '\n', body.group(1))
    text = re.sub(r'<p[^>]*>', '\n', text)
    text = re.sub(r'</p>', '', text)
    text = re.sub(r'<[^>]+>', '', text)
    text = html.unescape(text)
```

### Pattern 2: JS-string embedded (`window.msg_title`)

Some shorter articles embed the ENTIRE content inside a JS string assignment.
The `<div id="js_content">` is empty — content is populated dynamically by JS.
The article text lives in `window.msg_title = '...'` with `\\n` for newlines
and `\\'` for escaped quotes.

```python
# Try Pattern 2 when js_content is empty
title_match = re.search(r"window\.msg_title\s*=\s*window\.title\s*=\s*'(.+?)'\s*\|\|\s*", content, re.S)
if not title_match:
    title_match = re.search(r"window\.msg_title\s*=\s*window\.title\s*=\s*'(.+?)';(?:\s*$|\s*\n)", content, re.S)
if title_match:
    article = title_match.group(1).replace('\\n', '\n').replace("\\'", "'").replace('\\\\', '\\')
    # First line is the title, rest is body
    lines = article.strip().split('\n')
    title = lines[0]
    body = '\n'.join(lines[1:])
```

The `msg_title` variable is also the source of the `<title>` tag — it's a single JS string
that contains the full article text. The regex MUST use `'` (single quotes) because WeChat
uses single-quoted JS strings for this value.

## Key Patterns

- **Title**: `var msg_title = "...";` — HTML-escaped, use `html.unescape()`
- **Description**: `var msg_desc = "...";`
- **Body (Pattern 1)**: `<div id="js_content">...</div>` — server-rendered HTML with inline tags
- **Body (Pattern 2)**: `window.msg_title = '...\\n...';` — JS-embedded, `\\n` for newlines
- **Images**: `data-src` attribute (not `src`) — WeChat lazy-loads images
- **Author**: `var nickname = "...";` in JS

## Pitfalls

- **Desktop UA is critical** — mobile/bot user agents trigger the verification wall even with `requests`
- **Pattern 2: `js_content` div exists but is empty** — don't assume failure; check `window.msg_title` as fallback
- **Pattern 2: use single quotes in regex** — WeChat uses `'...'` not `"..."` for `msg_title` JS strings
- **Pattern 2: `\\n` not `\n`** — the JS string literal has escaped newlines; use `.replace('\\n', '\n')`
- **Image URLs need transformation** — `data-src` values are relative (`//mmbiz.qpic.cn/...`), prepend `https:`
- **Don't over-engineer** — sophisticated tools (Crawl4AI, CamouFox) are overkill and actually worse for WeChat articles