A scraper that works on a laptop usually meets CI in one of two ways. Either someone pastes the proxy password into the workflow file “just to get it green”, or the pipeline runs a full live scrape on every push and quietly spends a month of bandwidth by Wednesday. Both are avoidable, and fixing them is mostly a matter of deciding, up front, which runs need a live proxy at all.
This tutorial covers where proxy credentials should live in CI, how to keep them out of logs, how to split tests so most runs never touch the network, and how to rotate a credential without a scramble. Examples use GitHub Actions and GitLab CI with Shifter’s residential gateway, but the structure carries over to any runner.
What goes wrong
Four failure modes account for nearly every CI credential incident with scrapers:
| Failure | How it happens |
|---|---|
| Credential committed | A .env file or a hardcoded proxy URL lands in the repository |
| Credential in logs | A debug line prints the proxy URL, or an HTTP client logs request headers |
| Credential exposed to untrusted code | A pull request from outside the team runs with secrets available |
| Bandwidth burned | Every push runs a live scrape against real sites |
The first three are security problems. The fourth is a cost problem that also makes the pipeline flaky, because live websites change and your build should not fail when someone else’s page does.
Step 1: store the credential as a secret, never in code
Your residential username and password are on the plan page in the panel. Store them as two CI secrets:
SHIFTER_USERNAME: the full username as shown, for examplecustomer-USERNAMESHIFTER_PASSWORD: the password
GitHub Actions. Add both under the repository’s Settings, Secrets and variables, Actions. GitHub redacts secret values from logs, and, with the exception of GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository. That second rule matters for public repositories: an outside contributor’s pull request cannot read your proxy password, but it also cannot run your live tests, which you will handle in step 3.
GitLab CI. Add both as CI/CD variables, mark them masked, and mark them protected so they are available only to pipelines on protected branches or tags. One gotcha is specific enough to call out: GitLab can only mask a value that is a single line of 8 characters or more. Shifter’s residential passwords can be shorter than that, and GitLab will refuse to mask them. The fix is to store the pair as one masked variable instead:
SHIFTER_PROXY_AUTH=customer-USERNAME:PASSWORDThat value is comfortably over 8 characters and uses only characters GitLab allows in masked variables. Split it on the last colon in your code.
Add .env to .gitignore in the same commit, so a local file never follows the credential into the repository.
Step 2: build the proxy URL in code, and never print it
Assemble the proxy URL at runtime from the environment, in one place, so targeting and session flags are added consistently and nothing else ever handles the raw password:
import os
GATEWAY = "p.shifter.io:443"
def shifter_credentials():
if "SHIFTER_PROXY_AUTH" in os.environ:
username, password = os.environ["SHIFTER_PROXY_AUTH"].rsplit(":", 1)
return username, password
return os.environ["SHIFTER_USERNAME"], os.environ["SHIFTER_PASSWORD"]
def proxy_url(country=None, session=None, ttl=None):
username, password = shifter_credentials()
if country:
username += f"-country-{country}"
if session:
username += f"-sid-{session}"
if ttl:
username += f"-ttl-{ttl}"
return f"http://{username}:{password}@{GATEWAY}"
def redact(url):
# Safe to log: keeps the flags, drops the password.
creds, host = url.rsplit("@", 1)
return f"{creds.rsplit(':', 1)[0]}:***@{host}"Log redact(url) if you need to see which flags a run used. Never log the URL itself.
Secret masking has a blind spot worth knowing about. It matches the stored value, so a transformed value slips through. Proxy authentication is sent as a Proxy-Authorization header containing the base64 encoding of username:password, and a debug log of request headers prints that encoding, which no CI masker will recognise. Keep header-level debug logging off in CI. If you must assemble a sensitive string that is not itself a secret, register it with GitHub’s ::add-mask:: command before anything can print it.
Pass secrets to your scraper as environment variables, not command-line arguments. GitHub’s own guidance is to avoid passing secrets between processes on the command line where possible. Arguments are easy to see in process listings and tend to end up in shell traces.
Step 3: split tests so most runs never need a proxy
This step removes most of the cost and most of the flakiness. Divide scraper tests into three tiers:
| Tier | What it checks | Needs a proxy | When it runs |
|---|---|---|---|
| Parser tests | Extraction logic against saved HTML fixtures | No | Every push and pull request |
| Live smoke test | A handful of real requests through the gateway | Yes | Main branch and a schedule |
| Full run | The actual scrape | Yes | Its own schedule, or a manual trigger |
Parser tests are the bulk of your coverage. Save real responses as fixture files, and test that your selectors pull the right fields out of them. They run in seconds, cost nothing, need no secrets, and fail only when your code is wrong. When a site changes its layout, save a fresh fixture and update the parser in the same pull request.
The live smoke test confirms that credentials, targeting and connectivity work, not that every page parses. Keep it to a few requests. It should skip, rather than fail, when the secret is absent, which is exactly the situation on a fork pull request:
import os
import pytest
import requests
from scraper.proxy import proxy_url
live = pytest.mark.skipif(
not (os.environ.get("SHIFTER_PASSWORD") or os.environ.get("SHIFTER_PROXY_AUTH")),
reason="no proxy credentials in this environment",
)
@live
def test_gateway_exits_in_requested_country():
url = proxy_url(country="de")
r = requests.get("https://ipinfo.io/json", proxies={"http": url, "https": url}, timeout=30)
assert r.status_code == 200
assert r.json()["country"] == "DE"The full run belongs on a schedule, not on push, so a merge does not trigger a production-sized scrape.
Step 4: one session per job, never shared
Sticky sessions pin a run to one exit IP, which is what you want for multi-step flows like pagination. Shifter’s session id is any string you choose, with a default lifetime of 120 seconds that ttl overrides. The documentation’s caution applies directly to CI: do not reuse a session id across concurrent workflows, because requests from different jobs landing on the same IP read as suspicious to most anti-bot systems.
CI already hands you a unique value per run. Use it, plus the job index if you run a matrix:
import os
run = os.environ.get("GITHUB_RUN_ID") or os.environ.get("CI_PIPELINE_ID", "local")
job = os.environ.get("JOB_INDEX", "0")
url = proxy_url(country="us", session=f"ci{run}j{job}", ttl=600)Keep the id alphanumeric, since the username uses dashes to separate flags. For requests that are independent of each other, leave the session out entirely and each request rotates to a fresh IP.
Step 5: check quota before a big run
A scheduled scrape that runs out of bandwidth halfway is worse than one that never started. Shifter’s Usage and Quota API returns what remains on a plan, so a preflight step can skip the run cleanly:
import os
import sys
import requests
MIN_GB = float(os.environ.get("MIN_REMAINING_GB", "5"))
r = requests.get(
f"https://shifter.io/api/v1/memberships/{os.environ['SHIFTER_MEMBERSHIP']}/usage",
params={"api_token": os.environ["SHIFTER_API_TOKEN"]},
timeout=30,
)
r.raise_for_status()
plan = r.json()["data"]
if plan["metered"] and plan["remaining_gb"] < MIN_GB:
print(f"Only {plan['remaining_gb']} GB left, resets {plan['resets_at']}. Skipping run.")
sys.exit(78)The API token is generated in the panel under Account, API Tokens. Store it as a secret like the password: it belongs to your account rather than to one workspace, so it reads usage for every workspace you are a member of. The endpoint is rate limited to 60 requests per minute, far more than a preflight needs.
If a run does exhaust the plan with Extra Traffic disabled, the gateway returns 509 Bandwidth Limit Exceeded. Treat that as a stop condition, not something to retry.
Step 6: fail fast on authentication errors
Retries are right for transient network errors and wrong for credential errors. A 407 Proxy Authentication Required means the username, password or a flag is wrong, and it will be just as wrong on the next attempt. In CI, a retry loop around a 407 turns a one-second failure into a ten-minute one with a confusing log. Make your client treat 407 as fatal, print the redacted proxy URL, and stop. The usual causes, and the order to check them in, are covered in fixing 407 proxy authentication errors.
A complete GitHub Actions workflow
name: scraper
on:
push:
pull_request:
schedule:
- cron: "0 6 * * *"
workflow_dispatch:
jobs:
parser-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -r requirements.txt
- run: pytest tests/parsers
smoke-test:
if: github.ref == 'refs/heads/main'
needs: parser-tests
runs-on: ubuntu-latest
env:
SHIFTER_USERNAME: ${{ secrets.SHIFTER_USERNAME }}
SHIFTER_PASSWORD: ${{ secrets.SHIFTER_PASSWORD }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -r requirements.txt
- run: pytest tests/live
full-run:
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
needs: smoke-test
runs-on: ubuntu-latest
concurrency: scraper-full-run
env:
SHIFTER_USERNAME: ${{ secrets.SHIFTER_USERNAME }}
SHIFTER_PASSWORD: ${{ secrets.SHIFTER_PASSWORD }}
SHIFTER_API_TOKEN: ${{ secrets.SHIFTER_API_TOKEN }}
SHIFTER_MEMBERSHIP: ${{ vars.SHIFTER_MEMBERSHIP }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -r requirements.txt
- run: python scripts/check_quota.py
- run: python -m scraper.runThe parser tests run everywhere with no secrets. The smoke test and full run only exist where secrets do. The concurrency group stops two scheduled runs from overlapping and sharing an IP budget. The membership ID is not secret, so it lives in a plain repository variable.
One adjustment is worth making: as written, the quota script’s exit code 78 fails the job. If you would rather see a skipped run than a red one, give the check step an id, have it write a flag to $GITHUB_OUTPUT, and gate the scrape step on that output.
Rotating the credential
Rotate on a schedule, and immediately whenever a secret might have leaked: a public log, a departing contractor, a forked workflow you are unsure about.
On Shifter, the account owner or a workspace Admin can generate a new residential password from the plan page in the panel. Viewer and Billing members cannot. The gateway picks up the new password right away, and the old one stops working at the same moment, so plan the order:
- Pause the scheduled workflow, or accept that a run in flight will fail with a 407.
- Generate the new password in the panel.
- Update the CI secret straight away.
- Update every other consumer of the same plan. The password belongs to the plan, not to one pipeline, so anything else using it breaks at the same moment.
- Re-run the smoke test to confirm.
Point 4 is why it pays to know where a plan’s credential is used before you need to rotate it. If several independent pipelines share one plan, a single rotation touches all of them.
The bottom line
Most of the work in running scrapers from CI is keeping the proxy out of runs that do not need it. Parser tests against fixtures cover the logic on every push, with no secrets and no bandwidth. A small live smoke test on main proves the credentials still work. The real scrape runs on a schedule, checks its quota first, uses a session id nobody else shares, and stops at the first authentication error instead of retrying it.
The credential itself lives in the CI secret store, gets assembled in one function, and never gets printed, including in base64. For longer-running collection, the operational side of watching a pipeline is covered in monitoring a web scraping pipeline, and spreading one across regions in residential proxy failover for multi-region pipelines. Client-side proxy setup for browser-based scrapers is in configuring residential proxies in Selenium and Playwright.