275 lines
8.0 KiB
Markdown
275 lines
8.0 KiB
Markdown
---
|
||
name: proxy-testing
|
||
description: "代理获取、测试、使用 — 免费代理列表抓取 + 付费代理提供商(Bright Data)API操作 + Playwright浏览器cookie注入绕过登录"
|
||
version: 2.0
|
||
---
|
||
|
||
# Proxy 代理获取与测试
|
||
|
||
涵盖免费代理和付费代理提供商（Bright Data 等）的获取、测试和使用。
|
||
|
||
---
|
||
|
||
## 1. 免费代理获取
|
||
|
||
### proxyscrape.com（推荐，免费无需 API key）
|
||
|
||
```bash
|
||
# SOCKS5 俄罗斯代理
|
||
curl -s "https://api.proxyscrape.com/v2/?request=displayproxies&protocol=socks5&timeout=5000&country=ru"
|
||
|
||
# HTTP 代理
|
||
curl -s "https://api.proxyscrape.com/v2/?request=displayproxies&protocol=http&timeout=5000&country=us"
|
||
|
||
# 全局所有协议
|
||
curl -s "https://api.proxyscrape.com/v2/?request=displayproxies&protocol=all&timeout=5000"
|
||
```
|
||
|
||
参数：
|
||
- `protocol`: http / https / socks5 / socks4 / all
|
||
- `timeout`: 代理超时(ms)，建议 5000-10000
|
||
- `country`: ISO 两字母代码（如 ru, us, jp），留空=全球
|
||
- `anonymity`: all / elite / anonymous / transparent
|
||
- `ssl`: all / yes / no
|
||
|
||
返回格式：纯文本，每行 `IP:PORT`
|
||
|
||
### 其他免费源
|
||
|
||
```bash
|
||
# free-proxy-list.net（需要 JS 渲染，用 DrissionPage 或 Playwright）
|
||
# geonode free API
|
||
curl -s "https://proxylist.geonode.com/api/proxy-list?country=RU&limit=50&page=1&sort_by=lastChecked&sort_type=desc"
|
||
```
|
||
|
||
### 注意：proxy-tools.com 端口不可用
|
||
|
||
cn.proxy-tools.com 的端口被 reCAPTCHA + 付费墙保护。用 proxyscrape.com 替代。
|
||
|
||
---
|
||
|
||
## 2. 连通性测试
|
||
|
||
```bash
|
||
# HTTP 代理
|
||
curl -s -m 8 --proxy http://IP:PORT https://api.ipify.org?format=json
|
||
|
||
# SOCKS5 代理（本系统用 --socks5 而非 --proxy socks5h）
|
||
curl -s -m 8 --socks5 IP:PORT https://api.ipify.org?format=json
|
||
|
||
# SOCKS4 代理
|
||
curl -s -m 8 --socks4 IP:PORT https://api.ipify.org?format=json
|
||
```
|
||
|
||
### Python 批量测试脚本
|
||
|
||
```python
|
||
import subprocess, json
|
||
|
||
def test_proxy(proto, addr, timeout=8):
|
||
"""测试代理连通性，返回 (ok, detail)"""
|
||
if proto == "socks5":
|
||
cmd = ["curl", "-s", "-m", str(timeout), "--socks5", addr, "https://api.ipify.org?format=json"]
|
||
elif proto == "socks4":
|
||
cmd = ["curl", "-s", "-m", str(timeout), "--socks4", addr, "https://api.ipify.org?format=json"]
|
||
else: # http
|
||
cmd = ["curl", "-s", "-m", str(timeout), "--proxy", f"{proto}://{addr}", "https://api.ipify.org?format=json"]
|
||
|
||
try:
|
||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout+3)
|
||
if r.returncode == 0 and r.stdout.strip():
|
||
try:
|
||
data = json.loads(r.stdout.strip())
|
||
return True, data.get("ip", "")
|
||
except:
|
||
return True, r.stdout.strip()[:50]
|
||
return False, "timeout/empty"
|
||
except subprocess.TimeoutExpired:
|
||
return False, "timeout"
|
||
except Exception as e:
|
||
return False, str(e)[:60]
|
||
```
|
||
|
||
---
|
||
|
||
## 3. 匿名度检测
|
||
|
||
```bash
|
||
curl -s -m 8 --proxy http://IP:PORT https://httpbin.org/headers
|
||
```
|
||
|
||
判断标准：
|
||
- **透明代理**：响应中有 `X-Forwarded-For` 头（暴露真实IP）
|
||
- **匿名代理**：有 `Via` 头但无 `X-Forwarded-For`
|
||
- **高匿名/精英**：两个头都没有
|
||
|
||
SOCKS5/SOCKS4 代理天然高匿名（不会添加额外 header）。
|
||
|
||
---
|
||
|
||
## 4. 实际浏览验证
|
||
|
||
```bash
|
||
curl -s -m 10 --proxy http://IP:PORT https://ya.ru -L | head -5
|
||
curl -s -m 10 --proxy http://IP:PORT https://www.amazon.com -L | head -5
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Bright Data 付费代理
|
||
|
||
Bright Data 提供三种使用方式，优先级从高到低：
|
||
|
||
### 5.1 /request API（推荐 — 不需要 IP 白名单）
|
||
|
||
```python
|
||
import urllib.request, json
|
||
|
||
API_KEY = "你的API_KEY"
|
||
payload = {
|
||
"zone": "zone9", # Zone 名称
|
||
"url": "https://httpbin.org/ip",
|
||
"format": "json",
|
||
"method": "GET",
|
||
"country": "us", # 可选：指定国家
|
||
}
|
||
req = urllib.request.Request(
|
||
"https://api.brightdata.com/request",
|
||
data=json.dumps(payload).encode(),
|
||
method="POST",
|
||
headers={
|
||
"Authorization": f"Bearer {API_KEY}",
|
||
"Content-Type": "application/json",
|
||
}
|
||
)
|
||
resp = urllib.request.urlopen(req, timeout=30)
|
||
result = json.loads(resp.read().decode())
|
||
# result = {"body": "...", "status_code": 200, "headers": {...}}
|
||
```
|
||
|
||
**特点：**
|
||
- 不需要 IP 白名单
|
||
- `zone47` 支持 `country` 参数（`us`, `kz`, `ru` 已验证）
|
||
- zone47 不指定 country 时随机分配国家
|
||
- 只能用于 Active 的 zone（Disabled 的 zone 返回 407 "zone not found"）
|
||
|
||
### 5.2 直接代理连接（需要 IP 白名单）
|
||
|
||
```
|
||
服务器: brd.superproxy.io:22225
|
||
用户名: brd-customer-<account_id>-zone-<zone_name>
|
||
密码: <zone_password>
|
||
```
|
||
|
||
如果机器 IP 未在白名单中，返回 HTTP 407。
|
||
|
||
### 5.3 /zone/info API（查询 zone 状态）
|
||
|
||
```bash
|
||
curl -s -X POST "https://brightdata.com/api/zone/info" \
|
||
-H "Authorization: Bearer <API_KEY>" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"zone":"zone9"}'
|
||
```
|
||
|
||
**注意：** API Key 权限不足时返回 403。仅 CP-admin 级别 key 可用。
|
||
|
||
---
|
||
|
||
## 6. Playwright Cookie 注入（绕过登录）
|
||
|
||
从浏览器开发者工具导出 cookies（Application → Cookies → Export）后，可通过 Playwright 注入：
|
||
|
||
```python
|
||
import json
|
||
from playwright.sync_api import sync_playwright
|
||
|
||
def parse_cookies(cookie_str: str) -> list:
|
||
"""解析 Chrome 导出的 cookie JSON 为 Playwright cookies"""
|
||
cookies = json.loads(cookie_str)
|
||
for c in cookies:
|
||
# 关键修复：domain 去除前导点
|
||
if c["domain"].startswith("."):
|
||
c["domain"] = c["domain"][1:]
|
||
# 处理 secure/httpOnly/sameSite 字段
|
||
c.setdefault("secure", False)
|
||
c.setdefault("httpOnly", False)
|
||
c.setdefault("sameSite", "Lax")
|
||
c.pop("size", None)
|
||
c.pop("session", None)
|
||
# 修复过期时间格式
|
||
if "expires" in c and isinstance(c["expires"], int) and c["expires"] == -1:
|
||
c.pop("expires")
|
||
return cookies
|
||
|
||
with sync_playwright() as p:
|
||
browser = p.chromium.launch(headless=True)
|
||
context = browser.new_context()
|
||
|
||
# 先访问目标域名，再注入 cookie（某些 cookie 需要 domain 确认）
|
||
page = context.new_page()
|
||
page.goto("https://targetsite.com")
|
||
context.add_cookies(parsed_cookies)
|
||
page.reload() # 此时已登录状态
|
||
```
|
||
|
||
**陷阱：**
|
||
1. Domain 不能有前导点（Chrome 导出格式是 `.domain.com`，Playwright 需要 `domain.com`）
|
||
2. 某些 cookie（如 `__Secure-` 前缀）需要 secure=True + HTTPS 页面
|
||
3. `sameSite` 必须是 `"Strict"`, `"Lax"`, `"None"` 三者之一
|
||
4. 必须先用 `page.goto()` 访问域名再 `add_cookies()`，否则某些严格 cookie 被拒绝
|
||
|
||
**使用场景：** Bright Data CP 2FA 绕过、ERP 系统登录跳过验证码、Google 账户绕过密码输入。
|
||
|
||
---
|
||
|
||
## 7. Playwright React 覆盖层（Overlay）处理技巧
|
||
|
||
当页面被 React onboarding/tour 覆盖层挡住时：
|
||
|
||
```python
|
||
# 方法1：Escape 键关闭
|
||
page.keyboard.press("Escape")
|
||
page.wait_for_timeout(1000)
|
||
|
||
# 方法2：点击关闭按钮
|
||
page.locator('button[class*="close"], button[aria-label*="close"], .onboarding-close, .tour-close').first.click()
|
||
|
||
# 方法3：强制点击覆盖层后面的元素（用 force=True）
|
||
page.locator('text=Enable').first.click(force=True, timeout=5000)
|
||
```
|
||
|
||
**陷阱：**
|
||
- React 动态渲染的按钮可能在 overlay 关闭后才出现（需 wait_for_timeout）
|
||
- `force=True` 可能触发 `detached from DOM` 错误（按钮被重新渲染）
|
||
|
||
---
|
||
|
||
## 8. 与 CamouFox 集成
|
||
|
||
```python
|
||
from camoufox.sync_api import Camoufox
|
||
|
||
with Camoufox(
|
||
headless=True,
|
||
proxy={"server": "socks5://IP:PORT"}
|
||
) as browser:
|
||
page = browser.new_page()
|
||
page.goto("https://ya.ru")
|
||
```
|
||
|
||
proxy 格式：
|
||
- HTTP: `{"server": "http://IP:PORT"}`
|
||
- SOCKS5: `{"server": "socks5://IP:PORT"}`
|
||
- 带认证: `{"server": "socks5://IP:PORT", "username": "u", "password": "p"}`
|
||
|
||
---
|
||
|
||
## 预期结果
|
||
|
||
| 代理类型 | 可用率 | 寿命 | 匿名度 |
|
||
|---------|--------|------|--------|
|
||
| 免费代理 | 5-6% | 小时级 | 中高 |
|
||
| Bright Data DC | ~100% | 分钟-小时 | 高 |
|
||
| Bright Data Residential | ~100% | 动态IP | 高 |
|