Automating Daily Keyword Position Monitoring with a SERP API

Automating rank tracking looks like a cron job wrapped around an API call, and for the first week it is. The problems arrive later: two runs that are not comparable because one went out at a different hour, a day missing from the series that nobody notices until a chart looks wrong, an alert that fires on noise, and a client asking why your number disagrees with theirs.

None of those are fetching problems. They are pipeline problems, and they are worth designing for before you have six months of data with holes in it. Here is the build.

Decide the measurement contract first

Before writing any code, fix the variables that make two measurements comparable, because a rank is meaningless without them: the location, the device, the language, and whether the search is personalised. Those choices become constants in your job, not per-run options.

The reason is that any drift in them shows up in your data as ranking movement that never happened. A keyword measured from one city on Monday and another on Tuesday will appear to move. The full reasoning is in measuring accurate keyword rankings, and the short version is that consistency matters more than absolute precision.

Write the contract down as a schema, because it is also what you will hand a client when they ask what your numbers mean:

# one row per keyword per market: this is the unit of measurement
TARGET = {
    "keyword":  "residential proxies",
    "country":  "de",
    "city":     "berlin",        # optional, only where local intent matters
    "language": "de",
    "device":   "desktop",
}

Fetch through a SERP API

You can build the collection layer yourself on proxies, or call an API that returns parsed results. For a daily monitoring job the API path removes the two maintenance burdens that actually consume time: keeping up with result-page layout changes, and running the collection infrastructure.

A SERP API takes the parameters above and hands back structured results, so your job is a request and a write rather than a fetch and a parser:

import requests, datetime as dt

def fetch_serp(t):
    r = requests.get("https://serp.shifter.io/v1", timeout=45, params={
        "api_key":  API_KEY,
        "q":        t["keyword"],
        "gl":       t["country"],      # market
        "hl":       t["language"],     # interface language
        "location": t.get("city"),
        "device":   t["device"],
    })
    r.raise_for_status()
    return r.json()

Check the current parameter names against the API documentation rather than trusting a blog post, since these evolve. The alternative, running collection yourself over residential exits, is covered in why accurate rank tracking requires residential proxies, and the trade-off is control versus maintenance.

Store the whole result, not just your position

The most common design mistake, and the one that is expensive to undo, is storing a single number per keyword per day.

Store the full ranked list of URLs, the presence and position of result features, and the raw payload. Three reasons. Your competitors’ movements are the context that makes your own movement interpretable. Feature changes, such as an AI overview appearing above the fold, change what a position is worth without changing the number. And you cannot retroactively collect a SERP from last Tuesday, so anything you did not store is gone.

CREATE TABLE serp_snapshot (
  id          BIGSERIAL PRIMARY KEY,
  keyword     TEXT NOT NULL,
  country     TEXT NOT NULL,
  city        TEXT,
  device      TEXT NOT NULL,
  captured_at TIMESTAMPTZ NOT NULL,
  results     JSONB NOT NULL,        -- full ranked list
  features    JSONB NOT NULL,        -- ai overview, local pack, shopping
  raw         JSONB                  -- keep it, storage is cheaper than regret
);
CREATE INDEX ON serp_snapshot (keyword, country, device, captured_at DESC);

Positions are then derived from snapshots rather than stored as the primary record, which means a parsing fix can be re-applied to history instead of only to future runs.

Schedule for comparability

Run at the same time each day, in a window you keep stable, because results shift through the day and a series collected at varying hours has variance you cannot attribute.

Spread the work across that window rather than firing every keyword at once. A burst is both harder on the source and more likely to be throttled, and spreading costs you nothing when the deadline is hours away. Add jitter between requests, cap concurrency, and let the run take an hour rather than four minutes.

import random, time

def run_daily(targets, window_seconds=3600):
    gap = window_seconds / max(len(targets), 1)
    for t in targets:
        snapshot = fetch_with_retries(t)
        store(t, snapshot)
        time.sleep(gap * random.uniform(0.7, 1.3))   # spread, with jitter

Distinguish “no change” from “no data”

This is the detail that separates a trustworthy series from a misleading one. If a fetch fails and you write nothing, a later reader sees a gap that looks identical to a day when nothing moved. Then a comparison against “yesterday” silently spans two days and reports movement that did not happen.

Record every attempt with an explicit outcome, and make downstream analysis refuse to compare across a gap:

def fetch_with_retries(t, attempts=3):
    for i in range(attempts):
        try:
            data = fetch_serp(t)
            if is_valid(data):                     # sanity-check the payload
                return {"status": "ok", "data": data}
            record_status(t, "invalid")            # parsed, but not a real SERP
        except requests.RequestException:
            record_status(t, "error")
        time.sleep(2 ** i * random.uniform(0.5, 1.5))
    return {"status": "failed", "data": None}      # written as a failure, not skipped

Validating the payload matters as much as catching exceptions, because a response can arrive successfully and still not be a usable result page, which is the general problem in detecting blocked or fake content.

Alert on movement that means something

A naive alert on any position change will fire constantly, and an alerting system nobody trusts is worse than none.

Three rules make alerts useful. Require a threshold proportional to position, since a move from 3 to 6 matters far more than 47 to 50. Require persistence, meaning two or three consecutive runs, because single-day bounces are normal. And alert on entering or leaving the first page separately, since that boundary has real traffic consequences.

Then add the one alert people forget: collection health itself. If a market silently stopped returning data three days ago, that is more urgent than any ranking change, and it is only visible if you are tracking run outcomes as first-class data, per monitoring a scraping pipeline.

Make the data explain itself

Two additions turn a rank table into something you can put in front of a client.

Annotate your own timeline: deployments, content publishes, migrations. Half of all “why did we drop” investigations end with a release that went out that week, and an annotated chart finds it immediately.

Compute portfolio-level movement, not just per-keyword positions, because a broad shift across many unrelated keywords is a different event from one page slipping. That is the volatility analysis in detecting SERP volatility and algorithm updates, and it is only possible because you stored the full result set rather than your own position.

Collect responsibly

Keep to public search results, respect each engine’s terms of service, and pace politely rather than treating rate limits as an obstacle. A daily monitoring job has no reason to be aggressive: the deadline is the next morning, so spreading the work is free.

The bottom line

The cron job is the easy part. Fix your measurement contract first, so location, device, language and personalisation are constants rather than accidental variables. Store the full result set and the raw payload, not just your position, because competitor context and result features are what make a number interpretable and you cannot go back for them. Run at a consistent time, spread across the window with jitter. Record failures explicitly so a gap can never be mistaken for stability. Alert on persistent, position-weighted movement plus collection health. Then annotate your own changes, so the chart answers the question people actually ask.

The fetching layer is a SERP API if you want parsed results without maintaining collection, or residential proxies with country and city targeting if you would rather run it yourself, with per-GB pricing that suits small, frequent checks.