---
name: 1688-cross-border-sourcing
description: Scrape 1688.com for cross-border e-commerce products using Firecrawl API, rank by suitability, and enrich with Temu/Amazon pricing estimates.
version: 1.0
tags: [1688, sourcing, cross-border, firecrawl, scraping]
---

# 1688 Cross-Border E-Commerce Product Sourcing

Scrape 1688.com for products suitable for cross-border e-commerce (Temu/Amazon/AliExpress), then rank and analyze them.

## ⚠️ Critical: 1688 Anti-Scraping Reality

1688.com (owned by Alibaba) has **extremely aggressive anti-bot measures**. Most scraping methods FAIL:

| Method | Category Pages | Detail Pages (offer/XXX) | Reason |
|--------|:---:|:---:|--------|
| Direct browser_navigate | ❌ | ❌ | Sandbox can't render 1688 JS |
| Jina Reader (r.jina.ai) | ❌ | ❌ | Blocked by Alibaba Cloud WAF |
| Scrapling StealthyFetcher | ❌ | ❌ | IP blocked by Alibaba Cloud |
| CamouFox (headless) | ❌ | ❌ | Server IP blocked (datacenter) |
| Free proxy + CamouFox | ⚠️ | ⚠️ | Page loads but login popup blocks |
| Google/Bing cache | ❌ | ❌ | JS-only challenge page returned |
| AliExpress reference URLs | — | ❌ | JS-only SPA, 2KB bootloader shell |
| **Firecrawl API** | ✅ | ❌ | Category/search listing pages work; **detail pages TIMEOUT** (2026-07-06 verified, 120s+ no response) |
| **Desktop CDP Chrome** | — | ✅ | **ONLY method for individual detail pages** — Chinese residential IP bypasses WAF |

**For category/search listing pages**: Use Firecrawl API.
**For individual detail pages (detail.1688.com/offer/...)**: Desktop CDP is the ONLY reliable method. Do NOT attempt Firecrawl, Jina, or direct HTTP — they all fail from non-Chinese IPs.

### Why CamouFox Alone Cannot Access 1688

1688's anti-bot runs on two independent layers (see skill:cloakbrowser "反爬两层防御模型"):

1. Browser fingerprint layer — CamouFox bypasses this naturally
2. IP reputation layer — Alibaba Cloud WAF rejects all known datacenter IPs outright. The response is a redirect to `bixi.alicdn.com/punish/...` with "Access denied" / "unusual traffic" message

Even with `geoip=True` and `humanize=True`, CamouFox on a server IP will be blocked at layer 2. The only way to browse 1688 interactively is CamouFox + Chinese residential proxy:

```python
with Camoufox(headless=True, geoip=True, humanize=True,
              proxy={"server": "http://residential-proxy:port", "username": "user", "password": "pass"}) as browser:
    page = browser.new_page()
    page.goto("https://www.1688.com", timeout=30000)
```

For scraping (no interactive browsing needed), Firecrawl remains the best option — it routes through its own proxy infrastructure.

## Step-by-Step Workflow

### 1. Scrape 1688 Category Pages via Firecrawl

Use Firecrawl API with `waitFor=5000ms` to extract product data. Focus on **category listing pages**, NOT search pages (search forces login).

```python
import requests

FIRECRAWL_API_KEY = "fc-5536880adfa44fb78a3bd9da8dd52b53"

# Recommended categories for cross-border sourcing:
CATEGORIES = {
    "日用百货": "https://s.1688.com/s/company_offer_page/0_-1_-1.html?keyword=跨境日用百货",
    "宠物用品": "https://s.1688.com/s/company_offer_page/0_-1_-1.html?keyword=跨境宠物用品",
    "厨房用品": "https://s.1688.com/s/company_offer_page/0_-1_-1.html?keyword=跨境厨房小工具",
    "美妆工具": "https://s.1688.com/s/company_offer_page/0_-1_-1.html?keyword=跨境美妆工具",
    "家居收纳": "https://s.1688.com/s/company_offer_page/0_-1_-1.html?keyword=跨境家居收纳",
    "运动户外": "https://s.1688.com/s/company_offer_page/0_-1_-1.html?keyword=跨境运动户外",
    "数码配件": "https://s.1688.com/s/company_offer_page/0_-1_-1.html?keyword=跨境数码配件",
}

resp = requests.post("https://api.firecrawl.dev/v1/scrape", json={
    "url": category_url,
    "formats": ["markdown"],
    "waitFor": 5000,
}, headers={
    "Authorization": f"Bearer {FIRECRAWL_API_KEY}",
    "Content-Type": "application/json",
})
markdown_content = resp.json()["data"]["markdown"]
```

### 2. Parse Product Data from Markdown

Use regex to extract structured product data from Firecrawl's markdown output:

```python
import re

# Price pattern
price_pattern = r'[¥￥]\s*(\d+\.?\d*)'
# Sales pattern: 成交X万+件 or 成交X+件
sales_pattern = r'成交(\d+\.?\d*)(万?)\+?件'
# Cross-border tags to detect
cross_border_tags = ['跨境', 'temu', '亚马逊', 'amazon', '专供', '爆款']
```

Products appear in markdown as sections with title, price, and sales info.

### 3. Score Products for Cross-Border Suitability

Composite scoring (0-100):

| Factor | Weight | Criteria |
|--------|--------|----------|
| Cross-border tag | 30% | Has "跨境/Temu/亚马逊" in title = full points |
| Sales volume | 25% | 10万+ = 25, 1万+ = 20, 1000+ = 15, <1000 = 10 |
| Price point | 25% | ¥2-15 optimal range (cheap enough for margin) |
| Shipping friendliness | 20% | Light (<200g) + small package = full points |

### 4. Enrich with Cross-Border Pricing Estimates

For each selected product, estimate:

- **TEMU price**: 1688 price × 2.5-4x (USD), capped at typical TEMU range
- **Amazon price**: 1688 price × 4-6x (USD), capped at typical Amazon range
- **Margin**: (Selling price - 1688 cost - shipping) / Selling price
- **Typical shipping cost**: Small packet <500g ≈ ¥15-25 to US/EU

### 5. Best Product Categories for Cross-Border

Based on analysis, these categories consistently perform well:

1. **宠物用品** (Pet supplies) — High demand, emotional buying, good margins
2. **厨房小工具** (Kitchen gadgets) — Lightweight, impulse buy, proven sellers
3. **美妆工具** (Beauty tools) — Tech premium, high Amazon prices
4. **家居收纳** (Home organizers) — Low cost, high utility, evergreen
5. **日用百货** (Daily goods) — Volume play, staple items

## Desktop CDP Browser — PRIMARY method for individual detail pages

## ⚠️ SG5 Server-Side Scraping: ALL Methods Blocked (2026-07 Verified)

SG5 server IP (43.160.244.125) is blocked by Alibaba Cloud WAF at the IP layer. These ALL fail:

| Method | Result | Detail |
|--------|--------|--------|
| Direct HTTP (curl/urllib) | ❌ | Redirect to `bixi.alicdn.com/punish/...` |
| Jina Reader (r.jina.ai) | ❌ | "Access denied" page |
| Firecrawl API | ⚠️ | Works ~50% but times out on JS-heavy detail pages |
| Google/Bing cache | ❌ | Returns search pages, not cached product data |
| AliExpress pages | ❌ | SPA only (2KB JS bootloader), no server-rendered data |
| AliExpress API endpoints | ❌ | x5sec anti-bot challenge on all endpoints |

**The ONLY reliable path**: Desktop CDP Chrome (Chinese residential IP) → Bridge → scrape detail pages.

## Primary: Desktop CDP Chrome Scrape (Chinese IP)

The user's Desktop runs Chrome on a Chinese residential IP, bypassing Alibaba Cloud's datacenter IP block entirely.

**Automated scraping script**: `~/.hermes/scripts/1688_scrape_cdp.py`
- Polls Bridge `/health` for Desktop slot with `cdp_browser_running`
- Navigates to each 1688 offer URL via `/cdp/navigate`
- Extracts title, images, price, specs via `/cdp/evaluate`
- Writes enriched data to MongoDB `premiumproducts.products`
- Deploy via cron: `cronjob create no_agent=true script=1688_scrape_cdp.py schedule="every 5m"`

## Product Detail Enrichment Pipeline (Submit → Full Data)

For products submitted via Desktop Submit:

| Step | What | Tool |
|------|------|------|
| 1. Scrape | Extract raw data from 1688 page | CDP browser_console |
| 2. Structure | LLM extracts title/price/images/specs | Qwen 35B (:40006) / qwen3-vl |
| 3. Translate | zh → en, zh → ru | Qwen 35B / API |
| 4. Images | Remove watermark, resize 800×800 | SDXL/FLUX + AtomK toolbox |
| 5. Classify | Auto-categorize product | LLM |
| 6. Write | Update MongoDB document | Server endpoint |

## Reference Files

- `references/cdp-detail-extraction.md` — Extract structured product data from 1688 detail pages via Desktop CDP (body.innerText, image URL extraction, expression filter workarounds)
- `references/image-download-conversion.md` — Download alicdn product images and convert WebP→JPG in Cloud Bridge sessions (cron no_agent pattern)
- `references/mongodb-product-update.md` — Update product pool data via direct MongoDB connection when REST API is read-only (credentials, field schema, pitfalls)

## Scripts

- `scripts/1688_scrape_cdp.py` — Automated 1688 detail page scraper: health check → find Desktop slot → navigate → extract title/price/images/specs/weight → write MongoDB. Designed for cron `no_agent=true` polling: silently exits when no Desktop connected, runs when Desktop comes online.

## Pitfalls

- **Do NOT use search pages** (s.1688.com with keyword search forces login popup)
- **Category pages work better** — products visible without login
- **Product detail links** (offer/XXXX format) use encrypted redirects, hard to extract direct URLs
- **CamouFox on server IP = still blocked** — 1688 blocks at IP layer (datacenter detection), not browser fingerprint. CamouFox alone is insufficient; must pair with residential proxy or use Firecrawl
- **Free proxies are unstable** — may work one moment, blocked the next
- **Firecrawl waitFor=5000** is critical — without it, 1688's lazy-loaded content won't render
- **🔴 Firecrawl detail page TIMEOUT (2026-07-06 verified)**: Firecrawl API times out (120s+) on `detail.1688.com/offer/XXX` URLs. It works for category/search listing pages only. For individual detail pages, use Desktop CDP (Chinese IP).
- **🔴 AliExpress reference URLs are JS-only SPAs**: `aliexpress.com/item/XXX.html` returns a 2KB JS bootloader with no product data. The internal API (`aeglodetailweb/api/header`) also blocks non-browser requests with x5sec protection. AliExpress cannot be used as a fallback data source from server-side code.
- **🔴 Desktop must be connected for CDP scraping**: When Desktop is not running (0 slots on Bridge /health), CDP scraping is unavailable. Use a cron polling script (`no_agent=true`) that silently exits when no Desktop is connected, and runs the scraper when it comes online. See `scripts/1688_scrape_cdp.py` for the canonical pattern.
- **Firecrawl API key can expire** — returns `{"success": false, "error": "Unauthorized: Invalid token"}`. Fall back to Desktop CDP browser (Chinese IP) or renew the key.
- **Server-side CDP browser gets blocked** — `browser_navigate` to 1688 from SG3/SG5 returns `bixi.alicdn.com/punish/...` (Access Denied). Only Desktop Chrome on Chinese residential IP works for CDP-based scraping.
- **MOQ info** not reliably available from category pages; need detail page access for that
- **Prices shown may be tier prices** — the lowest price often requires high MOQ
- **Sales figures** on 1688 can be inflated; treat as directional, not exact

## Output Format

Save results as:
- `/tmp/1688_cross_border_top10.json` — Full enriched data
- `/tmp/1688_cross_border_top10.csv` — Spreadsheet-friendly format

CSV columns: 排名, 品类, 产品名称, 1688价格(元), 1688销量, 预估重量, TEMU参考价($), Amazon参考价($), 预估毛利率, 跨境标签, 推荐指数

## Related Skills

- `web-scraping-toolkit` — General scraping tools reference (Firecrawl is one of 5 tools documented there)
- `atomk-api-export` — Alternative product sourcing from AtomK system
- `temu-pipeline` — Downstream TEMU order processing