How to Use Residential Proxies in Go with net/http and Colly

Go is a natural fit for scraping: cheap concurrency, a solid standard library, and a single static binary to deploy. Wiring a residential proxy into net/http is genuinely three lines. The part that costs people a day is everything after those three lines, because Go’s HTTP client has a few sharp edges that silently destroy connection reuse, and through a proxy that turns into latency and timeouts you’ll struggle to explain.

This is the Go entry in the same series as residential proxies with Python and with Playwright: the working code, plus the specific traps that matter in Go.

Everything below uses the Shifter residential gateway: one endpoint, p.shifter.io:443, with all targeting encoded in the username. Swap host and credentials for a different provider; the shape is the same.

The gateway model in one paragraph

The proxy username carries your auth and your targeting. You don’t switch endpoints to change country or session, you change the username string:

customer-USERNAME-country-us-sid-abc123-ttl-600

country-us targets the US, sid pins a sticky session, ttl holds that IP for N seconds. Omit sid/ttl and each new connection rotates. Password is constant.

Unlike Chromium (which ignores inline proxy credentials), Go’s net/http handles credentials in the proxy URL correctly, sending Proxy-Authorization for you. So http://user:pass@host:port just works.

The minimal version

package main

import (
  "fmt"
  "io"
  "net/http"
  "net/url"
  "os"
  "time"
)

func main() {
  user, pass := os.Getenv("SHIFTER_USER"), os.Getenv("SHIFTER_PASS")
  proxyURL, err := url.Parse(fmt.Sprintf("http://%s-country-us:%s@p.shifter.io:443", user, pass))
  if err != nil {
    panic(err)
  }

  client := &http.Client{
    Transport: &http.Transport{Proxy: http.ProxyURL(proxyURL)},
    Timeout:   30 * time.Second,
  }

  resp, err := client.Get("https://api.ipify.org")
  if err != nil {
    panic(err)
  }
  defer resp.Body.Close()

  body, _ := io.ReadAll(resp.Body)
  fmt.Println(string(body)) // a US residential IP
}

That’s the whole integration. Now the parts that decide whether it performs.

Gotcha 1: reuse the client, never create one per request

This is the most common and most expensive Go mistake, and a proxy makes it worse.

http.Client and http.Transport are designed to be long-lived and shared. They’re safe for concurrent use by multiple goroutines, and the connection pool lives in the Transport. Create a new client per request and every request pays a fresh TCP handshake plus a TLS handshake through the proxy, which is exactly the overhead the latency guide says to eliminate. Worse, you leak idle connections.

// BAD: a new client per request. Zero connection reuse, leaks connections.
func fetch(u string) (*http.Response, error) {
  client := &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(proxyURL)}}
  return client.Get(u)
}

// GOOD: one client, package-level or injected, shared across goroutines.
var client = &http.Client{ /* configured once, see below */ }

Build it once at startup and pass it around.

Gotcha 2: drain and close the body, or you get no reuse

Go will only return a connection to the pool if the response body is fully read and closed. Close it without reading, and the connection is discarded, so you silently lose pooling even though your client is shared.

resp, err := client.Get(u)
if err != nil {
  return err
}
defer resp.Body.Close()

// If you don't need the body, still drain it so the connection can be reused.
// Deferred calls run last-in-first-out, so this drain runs *before* the Close
// above, which is the order you want.
defer io.Copy(io.Discard, resp.Body)

If you’re parsing the body normally (io.ReadAll, a JSON decoder that reads to EOF), you’re fine. The trap is early returns: bailing out on a non-200 status without draining leaves the connection unusable, and on a scraper that hits a lot of blocks, that’s most of your traffic.

Gotcha 3: MaxIdleConnsPerHost defaults to 2

This one bites every Go scraper. http.Transport’s default MaxIdleConnsPerHost is 2. Run 50 goroutines against one host and 48 of them keep opening and discarding fresh connections, each paying a full handshake through the proxy.

Set it to at least your per-host concurrency:

transport := &http.Transport{
  Proxy:               http.ProxyURL(proxyURL),
  MaxIdleConns:        200,
  MaxIdleConnsPerHost: 50,  // >= your per-host concurrency
  IdleConnTimeout:     90 * time.Second,

  // Granular timeouts beat one blunt Client.Timeout.
  TLSHandshakeTimeout:   10 * time.Second,
  ResponseHeaderTimeout: 30 * time.Second,
  ExpectContinueTimeout: 1 * time.Second,
}

client := &http.Client{Transport: transport}

Note there’s no Client.Timeout here on purpose. Client.Timeout is a single total budget covering dial, TLS, headers, and body. The granular transport timeouts let you tell a slow connect apart from a slow response, which is exactly the connect-versus-read distinction that makes timeouts diagnosable. Use context.WithTimeout per request for an overall deadline.

Rotating geo and sessions

Because targeting lives in the username, a different identity means a different proxy URL. Two approaches:

Per-request, via the Proxy func. Transport.Proxy is called per request, so you can vary the identity there. Go pools connections per proxy URL, so this stays efficient:

func proxyFor(country, sid string) (*url.URL, error) {
  u := fmt.Sprintf("%s-country-%s", os.Getenv("SHIFTER_USER"), country)
  if sid != "" {
    u += fmt.Sprintf("-sid-%s-ttl-600", sid)
  }
  return url.Parse(fmt.Sprintf("http://%s:%s@p.shifter.io:443", u, os.Getenv("SHIFTER_PASS")))
}

// Read targeting off the request context so each job picks its own identity.
type ctxKey string
const identityKey ctxKey = "identity"

type identity struct{ Country, SID string }

transport := &http.Transport{
  Proxy: func(r *http.Request) (*url.URL, error) {
    if id, ok := r.Context().Value(identityKey).(identity); ok {
      return proxyFor(id.Country, id.SID)
    }
    return proxyFor("us", "")
  },
  MaxIdleConnsPerHost: 50,
}

Then attach an identity per job:

ctx := context.WithValue(req.Context(), identityKey, identity{Country: "de", SID: "job-42"})
resp, err := client.Do(req.WithContext(ctx))

Or a transport per identity, cached in a map, when you have a small fixed set (one per country, say). Either works; the context approach scales better when identity is per work-item.

Give each logical unit of work its own sid and rotate between units rather than mid-flow (sticky vs rotating covers when each applies, and the load balancing post covers mapping work to identities).

Concurrency, per host

Go makes it trivial to launch 10,000 goroutines, and equally trivial to get yourself blocked. Bound concurrency per target host, not globally, so a fragile target can’t be hammered and a permissive one isn’t throttled:

import "golang.org/x/sync/semaphore"

var limits = map[string]*semaphore.Weighted{
  "tough-site.example": semaphore.NewWeighted(4),
  "open-site.example":  semaphore.NewWeighted(32),
}

func fetch(ctx context.Context, host, u string) error {
  sem := limits[host]
  if err := sem.Acquire(ctx, 1); err != nil {
    return err
  }
  defer sem.Release(1)
  // ... do the request
  return nil
}

Past a target’s tolerance, more parallelism buys blocks, not throughput (how to avoid getting blocked).

Colly

Colly is the standard Go scraping framework, and it takes a proxy in one line:

c := colly.NewCollector(colly.AllowedDomains("example.com"))

proxyURL := fmt.Sprintf("http://%s-country-us:%s@p.shifter.io:443", user, pass)
if err := c.SetProxy(proxyURL); err != nil {
  log.Fatal(err)
}

For per-request geo or sticky sessions, use SetProxyFunc, which is the same idea as the Transport.Proxy func above:

c.SetProxyFunc(func(r *http.Request) (*url.URL, error) {
  // Pick country/session per request, e.g. from the URL or a job map.
  return proxyFor("de", "job-42")
})

Colly has its own per-domain limiter, use it rather than rolling your own:

c.Limit(&colly.LimitRule{
  DomainGlob:  "*",
  Parallelism: 8,
  Delay:       200 * time.Millisecond,
  RandomDelay: 200 * time.Millisecond, // jitter, so requests aren't a metronome
})

And set a custom transport when you want the pooling and timeout settings from earlier:

c.WithTransport(transport)

Colly’s OnError is where retry logic belongs. Classify before retrying, and change identity on blocks rather than retrying the same exit, the classification table in the load balancing post applies directly.

Verify you’re actually on the proxy

Before benchmarking or debugging anything else, confirm the exit IP:

resp, _ := client.Get("http://ip-api.com/json")
defer resp.Body.Close()
b, _ := io.ReadAll(resp.Body)
fmt.Println(string(b)) // expect a residential IP in the targeted country

Your own IP means the client isn’t using the proxy. A hang means local egress is blocked. Both are covered in the timeout diagnostic guide.

FAQ

Does Go support proxy credentials in the URL? Yes. Unlike Chromium-based browsers, Go’s net/http handles http://user:pass@host:port properly and sends Proxy-Authorization for you. Since the gateway encodes targeting in the username, that’s all you need.

Why is my Go scraper slow through a proxy even with one client? Most likely you’re not draining response bodies (so connections never return to the pool), or MaxIdleConnsPerHost is still the default 2 while you run many goroutines. Fix both and per-request handshake overhead largely disappears.

How do I rotate IPs per request in Go? Vary the proxy username. Either return a different URL from Transport.Proxy (reading the identity off the request context) or keep a transport per identity. Go pools connections per proxy URL, so per-request variation is still efficient.

Should I use Client.Timeout or transport timeouts? Both, for different jobs. Granular transport timeouts (TLSHandshakeTimeout, ResponseHeaderTimeout) let you distinguish a slow connect from a slow response; use context.WithTimeout per request for the overall deadline. A single blunt Client.Timeout hides which phase failed.

Can I use http.ProxyFromEnvironment instead? Yes, it reads HTTP_PROXY/HTTPS_PROXY/NO_PROXY, which is handy in containers. It’s a fine default when every request uses the same identity, but you can’t vary geo or session per request that way, so use an explicit Proxy func when you need targeting.

The bottom line

Go plus residential proxies is a strong combination once you respect the standard library’s rules: share one long-lived client, always drain and close bodies, raise MaxIdleConnsPerHost to match your concurrency, and prefer granular transport timeouts over one blunt total. Vary the proxy username to change geo or session, bound concurrency per host, and let Colly’s limiter and SetProxyFunc do the same work at the framework level.

Get those right and the three-line integration performs the way Go should. Point it at the residential gateway, and remember pool quality decides how often you’re retrying at all (IP reputation). The pricing page has the per-GB plans to test it against your own targets.