How to Scrape Sites That Require Login: Sessions, Cookies, and Sticky Proxies at Scale

A lot of the data worth collecting sits behind a login, and scraping it is a different discipline from scraping public pages. The moment you authenticate, you stop being an anonymous visitor and start operating an account, and accounts get rate-limited, challenged, and banned in ways anonymous requests never do. The whole game shifts from “rotate freely so no single identity stands out” to “hold one identity steady so the account looks like a real, consistent user.”

One caveat before the how: only scrape data you are authorized to access, your own accounts, a partner’s data you have permission to pull, an API you are entitled to use. Logged-in data is a different legal and ethical category from public data, and whether the scraping is legal depends heavily on that line. This guide assumes you are on the right side of it.

Public scraping rotates. Authenticated scraping stays consistent.

The instinct from public scraping is to rotate everything, a new IP per request keeps any single identity from standing out. Behind a login, that instinct is exactly backwards. You now have a persistent identity, the account, and consistency is what makes it look legitimate. An account that logs in from fifty countries in an hour, or hops IPs mid-session, does not look like a power user. It looks compromised, and that is what triggers a lockout.

So authenticated scraping comes down to two things done well: managing the session so you stay logged in efficiently, and pinning each account to a stable, clean identity so it never looks like it teleported.

Managing the session

When you log in, the server hands back session state, usually cookies, sometimes a token. The single biggest mistake is re-logging-in on every request. Re-authentication is slow, and a flood of logins is itself a red flag that login endpoints rate-limit hard. Log in once, capture the session, and reuse it.

With a plain HTTP client, that means a persistent session object that keeps the cookie jar and rides the proxy:

import requests

proxies = {"https": "http://customer-USER-country-us-sid-acct42-ttl-600:PASS@p.shifter.io:443"}
s = requests.Session()
s.proxies.update(proxies)

# Log in once; the Set-Cookie response populates the jar.
s.post("https://example.com/login", data={"user": USER, "password": PW})

# Reuse the same session (and the same sticky IP) for every subsequent request.
r = s.get("https://example.com/account/data")

Two details matter beyond that. Many sites require a per-session CSRF or anti-forgery token that you scrape from a form or a prior page and send with write actions, so read it from the session rather than hardcoding it. And many sites, after login, expose a clean JSON API that the site’s own frontend calls; watching the network tab often reveals it, and hitting that authenticated API directly with your captured cookies or token is far faster and lighter than re-rendering pages.

Pin each account to one clean, sticky IP

This is where the proxy layer earns its place. An account should present a consistent location, so each account gets its own sticky session: it always exits through the same residential IP for the life of that session, encoded here as the sid in the username. Rotating IPs on a logged-in account is a classic ban trigger, because the account appears to jump between locations mid-session.

Three things make the identity hold up:

• Geo match. The IP should match where the account normally operates. A US account that suddenly appears on a German IP looks like a hijack, and many sites respond with a re-verification prompt or a lock.
• Clean reputation. Login endpoints scrutinize IP reputation harder than public pages, because that is where account takeovers happen. A flagged address gets extra friction, 2FA prompts, CAPTCHAs, “confirm it’s you” screens, before it even reaches the data.
• One account, one identity. If you run several accounts, each needs its own sticky IP, not a shared one. Accounts that all log in from a single address get linked and flagged together. This is the account-scale version of the two-layer identity an antidetect browser handles at the device level: distinct account, distinct IP, and if you use a browser, distinct fingerprint.

Scaling to many accounts

The pattern scales by mapping each account to its own stable identity and holding that mapping. Think of a registry: account to sticky sid, account to cookie jar, and, if you drive a browser, account to browser profile.

# One durable identity per account: same sid -> same exit IP, own cookie jar.
def session_for(account):
    s = requests.Session()
    sid = f"acct-{account['id']}"
    s.proxies.update({
        "https": f"http://{BASE_USER}-country-{account['geo']}-sid-{sid}-ttl-600:{PW}@p.shifter.io:443"
    })
    load_cookies(s, account)     # restore persisted jar, or log in if absent
    return s

Then spread load across accounts rather than pushing one account hard, and cap concurrency per account and per target, since each account has its own rate limit. The load-balancing principle applies, but the unit you distribute over is accounts, each on its fixed identity, not raw IPs.

Handle the failure modes

Three things go wrong, and each has a specific response.

Session expiry. Sessions time out. Detect it, a redirect to the login page, or a 401, and re-authenticate on the same identity, then resume. The critical rule is not to switch IP when you re-auth; a re-login from a new location is far more suspicious than the expiry itself.

Silent logout. Sometimes you get a 200 that is really the logged-out version of the page, all the public chrome, none of the account data. This is the authenticated cousin of a soft block: validate that you are still logged in by checking for an element only a signed-in user sees, rather than trusting the status code. If the account-only marker is gone, re-authenticate before you record empty rows.

Forced re-verification. A 2FA prompt or a “confirm it’s you” challenge usually means the identity looked risky, often because the IP was flagged or the location shifted. A clean, stable, geo-matched IP is what keeps these rare. When one does fire, treat that identity as under suspicion and back off rather than hammering through it.

Browser or HTTP for the login itself

Match the tool to the login, not the whole scrape. A simple form login that returns cookies works fine with a plain HTTP client, capture the cookies, reuse them, stay lightweight. But login pages are frequently the most heavily defended part of a site, with JS challenges, SSO redirects, dynamic tokens, and aggressive fingerprinting, precisely because that is where fraud happens. When the login resists a plain client, log in with a real browser (Playwright, Puppeteer, or Selenium), then either keep driving it or export the cookies to a lighter client for the bulk work. This is also where TLS and HTTP/2 fingerprinting bites hardest, so a real browser’s network fingerprint often makes the difference at the login step even when the rest of the scrape is fine on a simple client.

Verify before you trust a run

After authenticating, confirm two things: that you are actually logged in (fetch an account-only endpoint and check for a signed-in marker), and that you are exiting through the sticky IP you expect (an IP-echo check should return the same address for the life of the session). If the login “succeeded” but the account marker is missing, or the exit IP is drifting between requests, fix that before collecting anything, both are covered in the timeout and detection guides.

The bottom line

Scraping behind a login is about identity consistency, not rotation. Log in once and reuse the session rather than re-authenticating constantly, pin each account to one clean, geo-matched, sticky residential IP, keep one account to one identity, re-authenticate on the same identity when a session expires, and validate that you are still logged in instead of trusting a 200. Scale by adding accounts, each with its own stable identity, not by rotating a single account across addresses.

Get that right and authenticated collection is durable rather than a string of lockouts. A clean, sticky residential IP per account is the foundation the whole thing rests on, and the per-GB pricing lets you run many stable account identities and pay only for the data each one actually pulls.