Rotating Residential Proxies in Python: requests, Scrapy & Sticky Sessions

Python 轮转住宅代理：requests、Scrapy 与粘性会话

When you scrape at any real volume, the bottleneck is rarely your code — it’s the target site’s rate limiting and IP bans. Rotating residential proxies solve this by routing each request through a different real-user IP. Here’s how to wire them into requests and Scrapy in Python, including the sticky-session trick most tutorials skip. 当你进行大规模数据抓取时，瓶颈通常不在于代码，而在于目标网站的速率限制和 IP 封禁。轮转住宅代理通过将每个请求路由到不同的真实用户 IP 来解决此问题。以下是如何在 Python 的 requests 和 Scrapy 中配置它们，包括大多数教程忽略的“粘性会话”技巧。

The proxy URL format

代理 URL 格式

A residential proxy is just an authenticated HTTP/SOCKS endpoint. With a pool gateway, you target a country and control session behavior through the username, not separate endpoints: 住宅代理本质上是一个经过身份验证的 HTTP/SOCKS 端点。通过代理池网关，你可以通过用户名（而非单独的端点）来指定国家/地区并控制会话行为：

http://USERNAME_country-us_session-a1b2c3_lifetime-30:PASSWORD@proxy.gproxy.net:1000

country-us — exit country (ISO code)
session-a1b2c3 — a sticky-session id; reuse it to keep the same IP, change it to rotate
lifetime-30 — how many minutes that session’s IP stays fixed
country-us — 出口国家（ISO 代码）
session-a1b2c3 — 粘性会话 ID；重复使用它以保持相同的 IP，更改它则会轮转 IP
lifetime-30 — 该会话 IP 保持不变的分钟数

Basic request through a rotating proxy

通过轮转代理发送基础请求

import requests

USER = "USERNAME"
PWD = "PASSWORD"

def proxy(country="us", session=None, lifetime=30):
    tag = f"_country-{country}"
    if session:
        tag += f"_session-{session}_lifetime-{lifetime}"
    url = f"http://{USER}{tag}:{PWD}@proxy.gproxy.net:1000"
    return {"http": url, "https": url}

# New IP on every call (no session id):
r = requests.get("https://api.ipify.org?format=json", proxies=proxy(), timeout=30)
print(r.json()["ip"])

Run it in a loop and you’ll see a different IP each time — the gateway rotates automatically when no session id is present. 在循环中运行此代码，你会发现每次请求的 IP 都不同——当没有提供会话 ID 时，网关会自动进行轮转。

Sticky sessions: keep one IP across requests

粘性会话：在多个请求中保持同一个 IP

Some flows (login, multi-step checkouts, paginated results behind a cookie) break if your IP changes mid-session. Pin the IP by passing a stable session id: 某些流程（如登录、多步结账、基于 Cookie 的分页结果）如果中途更换 IP 就会中断。通过传递一个稳定的会话 ID 来锁定 IP：

import uuid
sess = uuid.uuid4().hex[:8] # one id for the whole flow
p = proxy(country="de", session=sess, lifetime=30)

s = requests.Session()
s.proxies.update(p)
s.get("https://example.com/login")
s.post("https://example.com/login", data={...}) # same exit IP

When you want a fresh IP, generate a new session id. 当你需要一个新的 IP 时，只需生成一个新的会话 ID 即可。

Scrapy integration

Scrapy 集成

Scrapy reads the proxy off each request’s meta. A tiny middleware rotates a fresh session per request: Scrapy 从每个请求的 meta 中读取代理信息。以下是一个简单的中间件，可为每个请求轮转一个新的会话：

# middlewares.py
import uuid

class ResidentialProxyMiddleware:
    USER = "USERNAME"
    PWD = "PASSWORD"

    def process_request(self, request, spider):
        sess = uuid.uuid4().hex[:8]
        cc = request.meta.get("country", "us")
        request.meta["proxy"] = (
            f"http://{self.USER}_country-{cc}_session-{sess}_lifetime-10:"
            f"{self.PWD}@proxy.gproxy.net:1000"
        )

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ResidentialProxyMiddleware": 350,
}

Set request.meta["country"] = "gb" in a spider to geo-target specific requests. 在爬虫中设置 request.meta["country"] = "gb" 即可对特定请求进行地理位置定位。

A few practical notes

一些实用建议

Retry on proxy errors. Residential IPs occasionally drop; wrap requests with retries (urllib3 Retry, or Scrapy’s RETRY_ENABLED) so a bad exit doesn’t kill a job. 在代理错误时重试。 住宅 IP 有时会掉线；请使用重试机制（如 urllib3 Retry 或 Scrapy 的 RETRY_ENABLED）包裹请求，以防因出口节点故障导致任务中断。
Respect the target. Rotating IPs is for spreading legitimate load and geo-testing — not for hammering a site past its terms. Add delays and honor robots.txt. 尊重目标网站。 轮转 IP 的目的是分摊合法的负载和进行地理位置测试，而不是违反服务条款去攻击网站。请添加延迟并遵守 robots.txt。
SOCKS5 works the same way on port 1002 if you need it (socks5://...). 如果需要，SOCKS5 在 1002 端口上的工作方式相同（使用 socks5://...）。

Getting credentials

获取凭据

You need a residential pool that supports country targeting and sticky sessions. I use GProxy — the username-based targeting above is exactly its format, and it’s usage-based ($0.90/GB) rather than a fixed monthly seat, which suits burst scraping jobs. Grab a key from the dashboard, drop it into USER/PWD, and the snippets above run as-is. 你需要一个支持国家定位和粘性会话的住宅代理池。我使用的是 GProxy——上述基于用户名的定位正是其格式，且它是按量计费（$0.90/GB）而非固定月费，非常适合突发性的抓取任务。从仪表板获取密钥，填入 USER/PWD，上述代码片段即可直接运行。

Happy (responsible) scraping. 祝抓取愉快（请负责任地抓取）。