100 lines
3.3 KiB
Markdown
100 lines
3.3 KiB
Markdown
---
|
|
name: proxy-management
|
|
description: "Proxy acquisition, testing, and management — free proxy lists, Bright Data commercial proxies, and browser cookie injection."
|
|
version: 1.0.0
|
|
author: Hermes Agent
|
|
license: MIT
|
|
metadata:
|
|
hermes:
|
|
tags: [proxy, bright-data, residential-proxy, socks5, scraping, free-proxy, commercial-proxy]
|
|
related_skills: [web-scraping-toolkit, camoufox-google-2fa-login]
|
|
---
|
|
|
|
# Proxy Management
|
|
|
|
Comprehensive guide for acquiring, testing, and using proxies — from free lists to commercial Bright Data residential/datacenter proxies.
|
|
|
|
## Choosing a Proxy Source
|
|
|
|
| Need | Source | Cost |
|
|
|------|--------|------|
|
|
| Quick one-off scraping | Free proxy lists (proxyscrape.com) | Free |
|
|
| Country/city targeting | Bright Data residential/DC | Paid |
|
|
| High reliability | Bright Data datacenter zones | Paid |
|
|
| Anti-bot bypass | Bright Data residential + browser | Paid |
|
|
|
|
## Section 1: Free Proxy Lists
|
|
|
|
### proxyscrape.com (recommended, no API key)
|
|
|
|
```bash
|
|
# SOCKS5 Russian proxy
|
|
curl -s "https://api.proxyscrape.com/v2/?request=displayproxies&protocol=socks5&timeout=5000&country=ru"
|
|
|
|
# HTTP proxies
|
|
curl -s "https://api.proxyscrape.com/v2/?request=displayproxies&protocol=http&timeout=5000&country=us"
|
|
```
|
|
|
|
Parameters: `protocol` (http/https/socks5/socks4/all), `timeout` (ms), `country` (ISO code), `anonymity` (elite/anonymous/transparent)
|
|
|
|
### Other Free Sources
|
|
|
|
- geonode API: `https://proxylist.geonode.com/api/proxy-list?country=RU&limit=50`
|
|
- free-proxy-list.net requires JS rendering (use DrissionPage or Playwright)
|
|
|
|
### Pitfalls
|
|
|
|
- Free proxies are unreliable — test before use, retry on failure
|
|
- cn.proxy-tools.com is behind reCAPTCHA + paywall — use proxyscrape.com instead
|
|
|
|
**See:** `references/proxy-testing.md` for full testing procedures and Playwright cookie injection.
|
|
|
|
## Section 2: Bright Data — Commercial Proxies
|
|
|
|
Bright Data provides residential, datacenter, ISP, and mobile proxy networks via API or control panel.
|
|
|
|
**Key endpoints:**
|
|
- Proxy: `brd.superproxy.io:22225` (non-SSL) or `brd.superproxy.io:33335` (SSL)
|
|
- API: `https://api.brightdata.com`
|
|
- Control panel: `https://brightdata.com/cp/zones`
|
|
|
|
### API Authentication
|
|
|
|
```python
|
|
import urllib.request, json
|
|
AK = "your-api-key"
|
|
|
|
def api_get(endpoint):
|
|
req = urllib.request.Request("https://api.brightdata.com" + endpoint)
|
|
req.add_header("Authorization", "Bearer " + AK)
|
|
resp = urllib.request.urlopen(req, timeout=15)
|
|
return json.loads(resp.read().decode())
|
|
```
|
|
|
|
### Zone Management
|
|
|
|
```bash
|
|
# List all zones
|
|
curl -s -H "Authorization: Bearer $AK" https://api.brightdata.com/zone | jq
|
|
|
|
# Enable/disable zones
|
|
curl -X PUT -H "Authorization: Bearer $AK" -H "Content-Type: application/json" \
|
|
-d '{"zone":"ZONE_NAME","enable":1}' https://api.brightdata.com/zone/set_enable
|
|
```
|
|
|
|
### Proxy Authentication Format
|
|
|
|
```
|
|
brd.superproxy.io:22225:zone-ZONE_NAME:PASSWORD
|
|
```
|
|
|
|
Country targeting: `-country-us` suffix in username. City targeting: `-city-us-newyork`.
|
|
|
|
### Pitfalls
|
|
|
|
- Zone passwords are separate from API key — retrieve via API or dashboard
|
|
- SSL endpoint (33335) required for HTTPS targets
|
|
- Residential proxies are metered — check GB usage before large scrapes
|
|
|
|
**See:** `references/brightdata-proxy-setup.md` for full API reference, zone management, and password rotation.
|