Java is a workhorse for data collection at scale: mature HTTP clients, real threads, and JVM tooling that makes long-running crawlers observable. Wiring a residential proxy into either of the two clients most teams use, OkHttp and Apache HttpClient, is straightforward. The friction is in the details each library handles differently: how proxy authentication is supplied, a connection-pool default that quietly throttles you, and the rule about consuming responses that decides whether pooling works at all.
This is the Java entry in the same series as residential proxies with Python, with Playwright, and with Go: the working code for both clients, plus the Java-specific traps.
Everything below uses the Shifter residential gateway: one endpoint, p.shifter.io:443, with all targeting encoded in the username. Swap host and credentials for a different provider; the shape is the same.
The gateway model in one paragraph
The proxy username carries your auth and your targeting. You don’t switch endpoints to change country or session, you change the username string:
customer-USERNAME-country-us-sid-abc123-ttl-600country-us targets the US, sid pins a sticky session, ttl holds that IP for N seconds. Omit sid/ttl and each new connection rotates. Password is constant. A wrinkle worth flagging up front: in both Java clients, proxy credentials are not put in the proxy URL, they go through a dedicated authentication mechanism. That’s the single most common thing people get wrong.
OkHttp
OkHttp takes a Proxy object for the address and a separate proxyAuthenticator for the credentials. Don’t try to encode user:pass@ in a URL; OkHttp won’t read it.
import okhttp3.*;import java.net.InetSocketAddress;import java.net.Proxy;import java.io.IOException;
public class ProxyExample { public static void main(String[] args) throws IOException { String user = System.getenv("SHIFTER_USER") + "-country-us"; String pass = System.getenv("SHIFTER_PASS");
OkHttpClient client = new OkHttpClient.Builder() .proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress("p.shifter.io", 443))) .proxyAuthenticator((route, response) -> { // Called when the proxy returns 407. Attach Proxy-Authorization. String credential = Credentials.basic(user, pass); return response.request().newBuilder() .header("Proxy-Authorization", credential) .build(); }) .build();
Request request = new Request.Builder().url("https://api.ipify.org").build(); try (Response response = client.newCall(request).execute()) { // try-with-resources closes the body System.out.println(response.body().string()); // a US residential IP } }}Two things to internalize. The user string includes the targeting flags (-country-us), because that’s where geo lives. And the try-with-resources around the Response is not optional style, it closes the response body, which is what returns the connection to the pool.
Reuse the client. OkHttpClient is designed to be created once and shared; it holds the connection pool and thread dispatcher, and it’s thread-safe. Creating one per request throws away pooling and leaks resources. To vary identity per request, derive a variant with newBuilder(), which shares the underlying pool and dispatcher:
// One base client, shared. Per-identity variants reuse its pool + dispatcher.OkHttpClient forGeo(OkHttpClient base, String country, String sid) { String user = System.getenv("SHIFTER_USER") + "-country-" + country + (sid != null ? "-sid-" + sid + "-ttl-600" : ""); String pass = System.getenv("SHIFTER_PASS"); return base.newBuilder() .proxyAuthenticator((route, resp) -> resp.request().newBuilder() .header("Proxy-Authorization", Credentials.basic(user, pass)) .build()) .build();}Give each logical unit of work its own sid and rotate between units, not mid-flow (sticky vs rotating covers the distinction).
Apache HttpClient (5.x)
Apache HttpClient supplies the proxy through the request config or a route planner, and credentials through a CredentialsProvider scoped to the proxy host.
import org.apache.hc.client5.http.classic.methods.HttpGet;import org.apache.hc.client5.http.impl.classic.*;import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager;import org.apache.hc.client5.http.auth.*;import org.apache.hc.client5.http.config.RequestConfig;import org.apache.hc.core5.http.HttpHost;import org.apache.hc.core5.util.Timeout;
public class ApacheProxyExample { public static void main(String[] args) throws Exception { HttpHost proxy = new HttpHost("http", "p.shifter.io", 443); String user = System.getenv("SHIFTER_USER") + "-country-us"; char[] pass = System.getenv("SHIFTER_PASS").toCharArray();
BasicCredentialsProvider creds = new BasicCredentialsProvider(); creds.setCredentials(new AuthScope(proxy), new UsernamePasswordCredentials(user, pass));
// Pool: raise per-route from the default of 5 (see gotcha below). PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(); cm.setMaxTotal(200); cm.setDefaultMaxPerRoute(50);
RequestConfig config = RequestConfig.custom() .setProxy(proxy) .setConnectTimeout(Timeout.ofSeconds(10)) // connect phase .setResponseTimeout(Timeout.ofSeconds(30)) // response phase .build();
try (CloseableHttpClient client = HttpClients.custom() .setConnectionManager(cm) .setDefaultCredentialsProvider(creds) .setDefaultRequestConfig(config) .build()) {
HttpGet get = new HttpGet("https://api.ipify.org"); // try-with-resources on the response consumes + releases the connection. try (var response = client.execute(get)) { System.out.println(new String(response.getEntity().getContent().readAllBytes())); } } }}Note the separate connect and response timeouts, that connect-versus-response split is exactly what makes timeouts diagnosable when something hangs.
Gotcha 1: Apache’s default max-per-route is 5
This is the Java equivalent of Go’s MaxIdleConnsPerHost trap, and it bites hard. PoolingHttpClientConnectionManager defaults to 2 connections per route and 20 total in older builds, and a low per-route cap in 5.x. Run 50 threads against one host and most of them block waiting for a connection to free up, which looks exactly like a slow proxy.
Set the pool to at least your per-host concurrency:
cm.setMaxTotal(200);cm.setDefaultMaxPerRoute(50); // >= your per-host concurrencyIf your crawler’s throughput plateaus no matter how many threads you add, this default is the first thing to check.
Gotcha 2: consume the entity, or the connection never returns
Both clients pool connections, and both only return a connection to the pool when the response is fully consumed and closed. In Apache HttpClient, an un-consumed entity keeps the connection checked out; do enough of that and your pool starves even though nothing is technically leaking.
Use try-with-resources on the response (as above), or explicitly consume:
import org.apache.hc.core5.http.io.entity.EntityUtils;// ...EntityUtils.consume(response.getEntity()); // drains + frees the connectionThe trap is early exits: bailing on a non-200 status without consuming the entity strands the connection. On a scraper that meets a lot of blocks, that’s most of your traffic, and the pool quietly dies.
Gotcha 3: reuse the client, always
Both OkHttpClient and CloseableHttpClient are heavyweight, thread-safe, and meant to be shared across the whole application. They own the connection pool, and a fresh one per request pays a full TCP + TLS handshake through the proxy every time, the overhead the latency guide exists to eliminate. Build one at startup, inject it, and derive request-scoped variants only when you must vary identity.
Rotating geo and sessions
Because targeting lives in the username, a different identity is a different credential string.
OkHttp: derive a client with newBuilder() per identity (shown above); it reuses the shared pool, so it stays efficient.
Apache HttpClient: attach a per-request HttpClientContext carrying a CredentialsProvider for that identity, so one client serves many identities without rebuilding:
HttpClientContext ctx = HttpClientContext.create();BasicCredentialsProvider perCall = new BasicCredentialsProvider();perCall.setCredentials(new AuthScope(proxy), new UsernamePasswordCredentials(userFor("de", "job-42"), pass));ctx.setCredentialsProvider(perCall);client.execute(get, ctx, resp -> { /* handle */ return null; });Rotate between logical units of work rather than within one, and map work to identities the way the load balancing post describes.
Concurrency, per host
Bound concurrency per target host, not with one global cap, so a fragile target can’t be hammered and a permissive one isn’t starved. A Semaphore per host is the simplest expression:
Map<String, Semaphore> limits = Map.of( "tough-site.example", new Semaphore(4), "open-site.example", new Semaphore(32));
void fetch(String host, Runnable work) throws InterruptedException { Semaphore sem = limits.get(host); sem.acquire(); try { work.run(); } finally { sem.release(); }}Past a target’s tolerance, more threads buy blocks, not throughput (how to avoid getting blocked). Size the pool’s per-route limit and your per-host semaphore together.
Verify you’re actually on the proxy
Before benchmarking or debugging anything else, confirm the exit IP:
Request req = new Request.Builder().url("http://ip-api.com/json").build();try (Response r = client.newCall(req).execute()) { System.out.println(r.body().string()); // expect a residential IP in the targeted country}Your own IP means the client isn’t using the proxy. A hang means local egress is blocked. Both are covered in the timeout diagnostic guide.
FAQ
Why doesn’t http://user:pass@host work for the proxy in Java?
Neither OkHttp nor Apache HttpClient reads inline proxy credentials from a URL. OkHttp uses a proxyAuthenticator that sets Proxy-Authorization; Apache uses a CredentialsProvider scoped to the proxy host. Since the gateway encodes targeting in the username, that username goes into the authenticator/credentials, not a URL.
My Java scraper stalls under load even with one client. Why?
Most likely the Apache pool’s per-route limit (low by default) is throttling you, or you’re not consuming response entities so connections never return to the pool. Raise setDefaultMaxPerRoute to your concurrency and consume every response with try-with-resources.
How do I rotate IPs per request in Java?
Vary the proxy username. In OkHttp, derive a per-identity client with newBuilder() (it shares the pool). In Apache HttpClient, pass a per-request HttpClientContext with a CredentialsProvider for that identity. Both let one shared client serve many identities.
Should I set connect and response timeouts separately? Yes. A separate connect timeout and response/socket timeout let you tell a slow connect (your side or no matching IP) apart from a slow response (the target), which is the key distinction when diagnosing timeouts. A single blunt timeout hides which phase failed.
OkHttp or Apache HttpClient for scraping? Either works well. OkHttp is lighter and has a cleaner API; Apache HttpClient is more configurable and long-established. Pick on ergonomics and existing dependencies; the proxy setup and the gotchas above apply to both.
The bottom line
Java plus residential proxies is solid once you respect each client’s rules: supply proxy credentials through the proper mechanism (OkHttp’s proxyAuthenticator, Apache’s CredentialsProvider), never inline in a URL; share one long-lived client and derive identity variants from it; raise Apache’s per-route pool limit to match your concurrency; always consume and close responses so connections return to the pool; and split connect from response timeouts. Vary the proxy username to change geo or session, and bound concurrency per host.
Get those right and both clients perform the way the JVM should. Point them at the residential gateway, and remember pool quality decides how often you’re retrying at all (IP reputation). The pricing page has the per-GB plans to test it against your own targets.