ADLAB logo
  • checklist
  • template

Rank error bars: when a position change is a drop, and when it is noise

One-page cheat sheet: error bars by position band, outlier rate and thresholds for jumps and retries. Each number has a date, a sample size and a limit.

  • v1.0
  • updated 12 November 2026
  • CC-BY-4.0
  • Vladislav Krivorutško
Download rank-error-bars.mdDownload all (.zip)Jump to the text

A rank tracker shows one number per keyword and per day, and since mid-September 2026 that number can come from a page that looks valid and is not. This sheet answers one question for each thing you can see in a tracker: is it a drop, or noise, and what do I do next. It gives error bars by position band, the share of outlier measurements, three thresholds (50 % shared domains for a "jump", 15 % for another retry round, 40 % as the cap on re-measures) and the traffic formula.

Every number carries a date, a sample size and a limit, and is marked as the author's own or a third party's. These are rules of thumb from small samples, not a standard. The own numbers were measured from 2026-11-10 to 2026-11-12, mostly on mobile pages for Estonia in the local language, through A-Parser and a proxy pool of the same country, against one reference browser. Section "Check it on your data" below takes 15 minutes.

Files, grouped by job

4 files · or all in one zip

The sheet

2 files

  • rank-error-bars.md

    Bands, outliers, third-party numbers, the decision table, thresholds and the traffic formula on one page

    10 KB

  • thresholds.csv

    Every threshold and number as a row: value, sample, date, own or third-party, limit, source

    5.0 KB

Try it first

1 file

  • example-readings.md

    One synthetic tracker day read with the rules

    1.7 KB

Read me

1 file

  • README.md

    3 steps and what to check before you rely on the numbers

    2.3 KB

Error bars by position band (own)

BandDomains shared with a browserRun-to-run shift of a site seen in both runs
Positions 1-3page 1: 70-90 %median 0, 90th percentile 4
Positions 4-10page 1: 70-90 %median 1, 90th percentile 7
Positions 11-20page 2: 50-75 %median 2, 90th percentile 5
Positions 21-30page 3: 10-44 %not measured
Positions 31-50pages 4-5: about 0 %not measured

Column 2 is one keyword against one browser (2026-11-10); the exact position matched in 7 to 16 % of cases, and two searches in the browser itself, one after the other, matched on 7 %. Column 3 is 30 neutral keywords run 4 times (6 pairs of runs, 2026-11-12): it tells how the tool agrees with itself, not how accurate it is, because these keywords have no browser reference.

Against a browser (6 keywords x 8 runs, 2026-11-12), a proxy pool of the same country gave a mean position error of 2.0 places inside the browser's top 5. The "plus or minus 2" in the video is that mean rounded; it is not an interval, and the 90th percentile of the shift in the top 3 is 4. The same test counted a measurement as good when it did not fail and at least 7 of the 10 domains matched the browser: 83 % (40 of 48; 95 % interval 70-91 %) for the same-country pool, 52 % for a pool of mixed countries and 15 to 44 % for hosted SERP APIs on default settings. One browser, one city.

How often one measurement is the wrong page (own)

On 58 keywords of one project (2026-11-11), 32 had 8 to 10 of 10 domains in common with a fresh measurement and 14 had 4 or fewer. The author's rough estimate is that 10 to 15 % of single measurements show the other version of the page. One site was shown 2nd for a keyword; in 22 of the next 24 measurements it was not in 2nd place, the live browser agreed, and its real place was about 20th. On the 4 x 30 neutral set, 44 of 624 cases (7 %) looked like a jump; in the 58-keyword sample it was 14 of 58 (24 %). The cheat sheet lists the rest, including the repeatability by page.

What I see, what I do

I seeI doNumbers
One keyword drops hard, one day, one measurementDo not report a drop. Measure again with the same settings. Under half of the top-10 domains shared with the last check means flag10-15 % of single measurements are the other page. DataForSEO also treats a one-day drop as unconfirmed
Two new measurements agree with each other, not with the old checkReal change: keep the new resultLogic of the re-measure rule
The second measurement looks like the old checkThe first was an outlier: keep the secondSimulation: 21 of 44 jumps took the second measurement, 23 kept the first
Cell is empty or cut offA failed measurement, not "lost". Retry only that keyword; a new round only if the last one brought back at least 15 %18 % of 613 keywords empty and 206 of 487 failed after one retry; after up to 3 rounds 1 to 3 failures in about 500
Two measurements in a row move the same wayTreat as confirmed, then compare the move with the bandDataForSEO: alerts on two consecutive readings
Move up to the band's median shiftIgnore. Top 3: up to 2 (mean error 2.0); 4-10: 1; 11-20: 2Section above
Move between the median and the 90th percentileWait for a second measurement before you write it down90th percentiles: top 3: 4; 4-10: 7; 11-20: 5
Move beyond the 90th percentileProbably real, or an outlier page; confirm before you actSame
A site at 3 goes to 6, another at 14 goes to 17Both are 3 places. The first is rarer, because the median shift is 0 in the top 3 and 2 at 11-20. One measurement proves neitherSection above
A site vanishes from positions 21-30Not a drop on one measurement: only 10-44 % of the domains of page 3 matched a browserSection above
A position beyond 30Do not use itPages 4-5 shared about 0 %
Many keywords flagged on one daySuspect the data source before each keyword; run again the next dayOwn flagged share 7 % on the neutral set and 24 % in the 58-keyword sample; third-party episodes 20-30 % of pages, and 0.09 % to 57.3 % of keywords in a day
Changes only at positions 1-5The poisoning study saw unrelated results from position 9 down and never in the top 5; look for another cause. My reading, not testedSE Ranking, US desktop

Thresholds

ThresholdValueWhere it comes from
Jump flagtop-10 domains shared with the last check: under 50 % of the smaller top 10The author's tracker rule
Pick of the second measurementkeep it only if it shares more domains with the last check than with the firstThe author's tracker rule; simulated on 624 cases
Retry of empty keywordsup to 3 rounds; next round only if the last one recovered at least 15 %Design choice, not fitted to data
Cap on re-measuresat most 40 % of the keywords of one checkDesign choice; keeps the proxy bill bounded
Page counted as a failureno organic result, or fewer than 8 per page while Google reports 200 or more resultsTracker defaults, not measured limits; see the A-Parser recipe
Depthtrust positions up to 30Pages 4-5 shared about 0 %

The simulation of the re-measure rule on 4 runs x 30 neutral keywords (624 cases): the match with a control run for the jumps rose from 0.62 to 0.86. The cases overlap, so it is an indication, not a statistic. The first live run had 9 keywords: 3 looked like jumps, 2 were re-measured, and both times the second result was taken. That is a smoke test.

Third-party numbers

NumberSourceLimit
Keywords with unrelated results: 0.09 % on 2026-09-16, 57.3 % on 2026-09-17. In 96.5 % of the cases the unrelated results began at position 9 or lower, none in the top 5SE Ranking study, published 2026-11-0310,000 keywords, US, desktop. They say "unrelated results" and do not claim intent
Poisoned page: fewer than half of the usual sites for the keyword. 20-30 % of collected pages since 2026-09-26; 2-3 % remain after two retries and a history check; noticeably fewer since 2026-10-10DataForSEO post, published 2026-10-23The provider's own data; no independent check. All large providers are affected, the cause is unknown
Option "Bypass poisoned SERP" in SE::Google; the vendor's test on two US pages (mobile and desktop) through residential proxies in New YorkA-Parser article, 2026-11-06One test by the vendor; how the option works is not described

Rather have this done for you?

Send the site address and what you want to know. We run the audit and send the report.

Send a request

Traffic formula

Cost = keywords x MB per keyword x price per GB / 1,000, in decimal GB. The author measured 1.0 to 1.7 MB per keyword at 3 pages; his own estimate uses 1.3. Proxy prices on 2026-11-12 ran from $1.70 per GB on small packs to about $1.00 on large ones. So 1,000 keywords cost $1.70 to $2.89 at $1.70 per GB, or $1.00 to $1.70 at $1.00. Re-measuring about 25 % of the keywords adds about 25 % traffic, which makes $2.13 to $3.61 at $1.70 per GB; the 40 % cap limits the extra to 40 %. For scale, a hosted API queue at depth 30 cost about $1.50 per 1,000 keywords on 2026-10-15 and again on 2026-11-12.

Check it on your data

  1. Pick 10 keywords where you know the result. In an incognito window with your language and country, write down the top 10.
  2. Run the same keywords through your tool 3 times.
  3. For each keyword count the domains shared by the top 10 of two runs. Under half means a flipped page: do not trust that day.
  4. Compare the average with your browser. If page 1 shares 70 % or more, the tool is good enough for trends.

Replace any threshold here that your numbers do not support. The counting is done by the SERP repeat check script, and the raw repeated measurements for comparison are in the repeatability dataset.

What it does not tell you

  • It is not a measure of truth. The reference is one browser of one person; Google itself changes the order of the first three results between two searches.
  • The samples are small: one keyword in the band test, 6 keywords in the browser comparison, 30 neutral and 58 project keywords elsewhere. The 52 % of the mixed-country pool includes a start-up loss of 13 of its first 18 measurements.
  • The hosted API numbers are from default settings and may have changed since October. DataForSEO reports fewer poisoned pages since 2026-10-10.
  • Estonia, mobile, local-language keywords. Other languages, countries and desktop pages were not tested for accuracy.
  • The 15 % and the 40 % are choices, not results.

Tested on

This is a sheet, not software. On 2026-11-12 every own number was compared with the author's dated measurement notes, and the SE Ranking and DataForSEO numbers were compared with the text of the two pages read the same day. The A-Parser article was read through a text extraction. The thresholds have a simulation on 624 cases and one live smoke test of 9 keywords behind them; no independent data set has confirmed them.

Files are provided as is: run them on your own data and check the result.

More files from the same audit

Want an audit like this on your site?

If you would rather have it done for you, send a request and describe the site.

Send a request
Download rank-error-bars.md