Design a Geo-Holdout Test Without Platform Help

We built a world in which advertising adds exactly 4.00% to sales. Then we ran a properly randomised geo test on it, with 20 matched pairs of regions over eight weeks. The test said +0.35%, and its 95% interval ran from -2.16% to +2.87%. The truth was outside the interval.

We kept that result. It is the first run of the script at the seed we fixed before looking, and it teaches more than a flattering one would. A good design is one that is right across many runs and tells you how wrong a single run may be. This piece shows how to build that design with regions and a spreadsheet.

Hypothetical demonstration: all data below is synthetic and generated by the code in this post.

Objective

Measure the incremental sales from a media programme using only things a marketing team controls: where the ads run, and a sales file by region. No platform lift tool, no user-level data and no consent-dependent tracking.

The business problem

Attribution reports who was present at the sale. They cannot say what would have happened without the ad. Blake, Nosko and Tadelis (2015) and Gordon et al. (2019) are the standard warnings: when the comparison is not randomised, advertising effects can look far larger than they are.

Platform lift tools answer the causal question, but the seller of the media runs them, and they usually cover one platform at a time. Geography is the neutral alternative. You can buy or stop media by region in most markets: Nielsen market areas in the US, states and metros in India, TV regions in the UK. Your own sales ledger records the result. Nothing in the method depends on a cookie.

The design

Forty regions, 26 weeks of history, eight test weeks. This is a go-dark holdout: ads continue in half the regions and stop in the other half. The script compares three ways of choosing the halves.

• Hand-picked: go dark in the 20 smallest regions, because that is the cheap and comfortable choice.
• Random: a straight draw of 20 from 40.
• Matched pairs: sort regions by past growth, pair neighbours in the list, toss a coin inside each pair.

The synthetic world has one deliberate trap. Larger regions grow faster, as they often do in real markets. The trap is invisible unless the design treats it properly.

Tools and methods

Python with numpy for the demonstration. For live use, a spreadsheet is enough. The estimate is a difference in growth rates, a form of difference-in-differences. Vaver and Koehler (2011) set out the geo experiment logic in a published form, and Kerman, Wang and Vaver (2017) give a time-based regression alternative that suits smaller panels.

Data you need

• Weekly sales by region for at least 26 weeks, from your own ledger rather than an ad platform.
• A region definition that your media can actually respect. If you can't switch a region off, it isn't a region.
• Known exceptions: store openings, stock-outs, local promotions.

The analytical framework

For each region, compute one number: =LN(AVERAGE(test weeks)/AVERAGE(pre weeks)). That is the region's growth. For each pair, subtract the dark region's growth from the lit region's. The effect is the average of those differences. The standard error is =STDEV.S(diffs)/SQRT(COUNT(diffs)). The interval is the effect plus or minus =T.INV.2T(0.05, pairs-1) times the standard error.

The logic is simple. A coin decided which region in each pair went dark, so nothing except the ads and luck separates the two groups. The interval is a statement about the luck.

Step by step

1. Define regions and export the sales history.
2. Sort by pre-period growth and form pairs. Drop or merge any region that dominates total sales.
3. Run a fake test inside the history before spending anything. Pretend the last eight weeks were a test, and repeat with many random assignments. The spread of those fake effects is your noise.
4. Multiply the noise by 2.8. That is the smallest effect you can expect to detect with about 80% power. If it exceeds any effect you could plausibly have, add regions or stop now.
5. Toss the coins. Write the assignment down and share it before the test starts.
6. Run the test. Change nothing else by region.
7. Compute pair differences, the interval and a sign-flip check, in which you randomly flip the signs of the pair differences many times and see how often the mean is as extreme as yours.

Metrics to examine

• Estimated lift and its interval, never the point alone.
• Bias, spread, coverage and power across simulated worlds, before the live test.
• False positive rate when the true effect is zero.
• Media leakage into dark regions, from reach data or a delivery log.
• Sales forgone. With equal-sized regions, a 4% effect and half the regions dark costs about 2% of sales for the test period, offset by the media saved.

The code

"""Geo-holdout test with regions and spreadsheet-level maths.
Hypothetical demonstration: every number is synthetic. Standard library + numpy only."""
import numpy as np

SEED, G, PRE, POST, RHO = 2024, 40, 26, 8, 0.5
EFFECT = 0.04                 # true: ads add 4% to sales in regions where they stay on
RUNS = 3000
TCRIT = {9: 2.262, 19: 2.093, 38: 2.024, 39: 2.023}   # t values for a 95% interval, by df

def world(rng, g=G, post=POST):
    """Log weekly sales: size + shared season + regional growth + AR(1) noise."""
    T = PRE + post
    size = np.sort(rng.normal(0, 1, g))                         # standardised region size
    growth = 0.0010 + 0.0012 * size + rng.normal(0, 0.0012, g)  # big regions grow faster
    noise_sd = 0.05 * np.exp(-0.25 * size)                      # small regions are noisier
    t = np.arange(T)
    season = 0.15 * np.sin(2 * np.pi * t / 52)
    e = np.zeros((g, T)); e[:, 0] = rng.normal(0, noise_sd)
    for k in range(1, T):
        e[:, k] = RHO * e[:, k - 1] + rng.normal(0, noise_sd)
    return 9 + size[:, None] + season + growth[:, None] * t + e, size

def change(y, a, b, c, d):
    """Per region: log(mean sales in window 2) minus log(mean sales in window 1)."""
    return np.log(np.exp(y[:, c:d]).mean(1)) - np.log(np.exp(y[:, a:b]).mean(1))

def assign(design, y, size, rng):
    """Return (on, pairs). on[g] is True if ads stay on; pairs is None if unpaired."""
    g = len(size)
    if design == "hand-picked":             # go dark in the smallest, cheapest regions
        on = np.zeros(g, bool); on[np.argsort(size)[g // 2:]] = True
        return on, None
    if design == "random":
        on = np.zeros(g, bool); on[rng.permutation(g)[: g // 2]] = True
        return on, None
    n = y.shape[1]                          # matched pairs on observed pre-period growth
    order = np.argsort(change(y, 0, n // 2, n // 2, n))
    pairs = order.reshape(-1, 2); on = np.zeros(g, bool)
    on[pairs[np.arange(len(pairs)), rng.integers(0, 2, len(pairs))]] = True
    return on, pairs

def estimate(r, on, pairs):
    """Difference in growth between on and off regions, with its standard error."""
    if pairs is None:
        a, b = r[on], r[~on]
        return a.mean() - b.mean(), np.sqrt(a.var(ddof=1) / len(a) + b.var(ddof=1) / len(b))
    d = np.where(on[pairs[:, 0]], r[pairs[:, 0]] - r[pairs[:, 1]], r[pairs[:, 1]] - r[pairs[:, 0]])
    return d.mean(), d.std(ddof=1) / np.sqrt(len(d))

def run(design, rng, effect=EFFECT, leak=0.0, g=G, post=POST):
    y, size = world(rng, g, post)
    on, pairs = assign(design, y[:, :PRE], size, rng)
    y = y.copy()
    y[on, PRE:] += effect                   # ads on: full effect
    y[~on, PRE:] += effect * leak           # holdout regions still catch some of the media
    r = change(y, 0, PRE, PRE, PRE + post)
    return estimate(r, on, pairs), (on, pairs, r)

def monte_carlo(design, seed, **kw):
    rng = np.random.default_rng(seed)
    out = np.array([run(design, rng, **kw)[0] for _ in range(RUNS)])
    est, se = out[:, 0], out[:, 1]
    target = kw.get("effect", EFFECT)
    crit = TCRIT[kw.get("g", G) // 2 - 1] if design == "pairs" else TCRIT[kw.get("g", G) - 2]
    cover = np.mean(np.abs(est - target) < crit * se)
    return est.mean(), est.std(), cover, np.mean(est - crit * se > 0)

def aa_forecast(rng):
    """Before spending: fake a test inside the pre-period and read the noise."""
    sds = []
    for _ in range(40):
        y, size = world(rng)
        y = y[:, :PRE]
        split = []
        for _ in range(50):
            on, pairs = assign("pairs", y[:, : PRE - POST], size, rng)
            r = change(y, 0, PRE - POST, PRE - POST, PRE)
            split.append(estimate(r, on, pairs)[0])
        sds.append(np.std(split))
    return float(np.mean(sds))

rng = np.random.default_rng(SEED)
(est, se), (on, pairs, r) = run("pairs", rng)
print("ONE TEST, MATCHED PAIRS (true effect +4.00%)")
print(f"  regions on/off: {on.sum()}/{(~on).sum()}, pairs: {len(pairs)}")
print(f"  estimate {est*100:+.2f}%   95% interval {100*(est-2.09*se):+.2f}% to {100*(est+2.09*se):+.2f}%")
d = np.where(on[pairs[:, 0]], r[pairs[:, 0]] - r[pairs[:, 1]], r[pairs[:, 1]] - r[pairs[:, 0]])
flips = rng.choice([-1, 1], size=(20000, len(d)))
print(f"  sign-flip randomisation p-value: {np.mean(np.abs((flips * d).mean(1)) >= abs(d.mean())):.4f}")

print(f"\nDESIGNS OVER {RUNS} SYNTHETIC WORLDS (true effect +4.00%)")
print(f"  {'design':<13}{'mean est':>10}{'bias':>8}{'sd':>7}{'95% cover':>11}{'power':>7}")
for i, dsg in enumerate(["hand-picked", "random", "pairs"]):
    m, s, c, p = monte_carlo(dsg, SEED + 1 + i)
    print(f"  {dsg:<13}{m*100:>9.2f}%{(m-EFFECT)*100:>+7.2f}%{s*100:>6.2f}%{c*100:>10.1f}%{p*100:>6.1f}%")

print("\nFALSE POSITIVES WHEN THE TRUE EFFECT IS ZERO (tests declaring a positive lift)")
for i, dsg in enumerate(["hand-picked", "random", "pairs"]):
    print(f"  {dsg:<13}{monte_carlo(dsg, SEED + 10 + i, effect=0.0)[3]*100:>6.1f}%")

print("\nMEDIA LEAKAGE: 30% of the effect reaches holdout regions (matched pairs)")
m, s, c, p = monte_carlo("pairs", SEED + 20, leak=0.3)
print(f"  mean estimate {m*100:.2f}% against a true +4.00%; power {p*100:.1f}%")

print("\nSIZE AND LENGTH (matched pairs, true effect +4.00%)")
print(f"  {'regions':>8}{'weeks':>7}{'sd':>8}{'power':>8}")
for g, post in [(20, 4), (20, 8), (40, 4), (40, 8), (40, 12), (80, 8)]:
    m, s, c, p = monte_carlo("pairs", SEED + 30 + g + post, g=g, post=post)
    print(f"  {g:>8}{post:>7}{s*100:>7.2f}%{p*100:>7.1f}%")

sd = aa_forecast(np.random.default_rng(SEED + 99))
print(f"\nA/A FORECAST FROM PRE-PERIOD ONLY: noise sd {sd*100:.2f}%, "
      f"minimum detectable effect (2.8 sd) {2.8*sd*100:.2f}%")

The output

ONE TEST, MATCHED PAIRS (true effect +4.00%)
  regions on/off: 20/20, pairs: 20
  estimate +0.35%   95% interval -2.16% to +2.87%
  sign-flip randomisation p-value: 0.7678

DESIGNS OVER 3000 SYNTHETIC WORLDS (true effect +4.00%)
  design         mean est    bias     sd  95% cover  power
  hand-picked       7.14%  +3.14%  1.44%      43.1%  99.9%
  random            4.01%  +0.01%  1.53%      94.7%  72.4%
  pairs             3.95%  -0.05%  1.47%      94.6%  73.0%

FALSE POSITIVES WHEN THE TRUE EFFECT IS ZERO (tests declaring a positive lift)
  hand-picked    57.2%
  random          2.6%
  pairs           2.6%

MEDIA LEAKAGE: 30% of the effect reaches holdout regions (matched pairs)
  mean estimate 2.78% against a true +4.00%; power 44.7%

SIZE AND LENGTH (matched pairs, true effect +4.00%)
   regions  weeks      sd   power
        20      4   2.31%   35.0%
        20      8   2.13%   43.8%
        40      4   1.59%   66.6%
        40      8   1.45%   74.1%
        40     12   1.41%   79.6%
        80      8   1.01%   97.1%

A/A FORECAST FROM PRE-PERIOD ONLY: noise sd 1.38%, minimum detectable effect (2.8 sd) 3.88%

Expected versus actual

Question	What the method predicts	What the run showed
Does randomisation remove bias?	Yes	Yes: bias +0.01% (random), -0.05% (pairs)
Do 95% intervals cover the truth 95% of the time?	Yes	94.7% and 94.6%. The single run above sits in the other 5%
Is hand-picking dangerous?	Yes, if regions differ in growth	Bias +3.14%, and 57.2% of tests declare a lift when the true effect is zero
Does matching add a lot of precision?	We expected it would	No: spread 1.47% against 1.53%, power 73.0% against 72.4%
Do longer tests help?	Somewhat	Less than regions do. Four to twelve weeks at 40 regions moved power from 66.6% to 79.6%. Going from 40 to 80 regions at eight weeks moved it from 74.1% to 97.1%
Does a fake test forecast the noise?	Yes	Forecast 1.38%, realised 1.47%. Close, slightly optimistic

Two results deserve attention. The hand-picked design has the best-looking power in the table, 99.9%, because it is measuring the growth gap between large and small regions and calling it advertising. A design can be almost certain to find something and still find the wrong thing.

The other result goes against our own recommendation. Matching barely helped here. In this world, most noise in a region's growth is fresh noise, not a persistent trait that pairing can cancel. Pairs remain cheap insurance against an unlucky draw and they make the spreadsheet easy, but the coin does the work. Past growth does not.

Limitations and bias

The world is ours. Effects begin at once and end at once, so there is no carryover, and a real go-dark test of eight weeks would miss part of what brand advertising does. A short test will tend to understate slow effects.

Leakage is the live threat. When 30% of the effect reached the dark regions, the mean estimate fell to 2.78% and power to 44.7%. Leakage always pushes the estimate toward zero, so a leaky test can make a working channel look weak.

Regions here count equally. If one metro holds a third of your sales, an equal-weighted estimate describes the average region and not the business. Commuters, national TV, online orders shipped across boundaries and regional stock problems all break the assumption that one region's ads do not touch another's sales.

Forty regions may not exist for you. With 20, eight weeks gave 43.8% power on a 4% effect. Run the fake test and accept the answer.

What to do with this

Treat the holdout as a question you put to the world, and size it before you ask. A team that tests with 40% power will read most of its results as "no effect" and retreat to the attribution report. The practical order is: choose the channel with the most spend and the least evidence, run the fake test, then fund the region count the noise demands.

Report the interval to the board, including when it is embarrassing. One test in twenty will miss the truth, as ours did. That is why a programme of repeated tests beats a single verdict.

The workflow

Sales ledger by region
        |
Region growth, weeks 1-26 --> sort --> pairs
        |
Fake test in history --> noise --> 2.8 x noise --> smallest detectable effect
        |                                               |
        |                         too large? add regions or stop
        v
Coin toss per pair (published before start)
        |
Eight test weeks, delivery log checked for leakage
        |
Pair differences --> mean, interval, sign-flip p-value
        |
Decision: does the interval clear the break-even lift?

Companion repository

The companion folder for this piece holds geo_holdout.py, the full script above, and output.txt, its saved output. It needs Python and numpy, takes about a minute, and reproduces the output with its fixed seed. Change EFFECT, G and POST first. Then replace world() with your own history.

Sources: Vaver and Koehler, "Measuring Ad Effectiveness Using Geo Experiments" (2011). Kerman, Wang and Vaver, "Estimating Ad Effectiveness Using Geo Experiments in a Time-Based Regression Framework" (2017). Blake, Nosko and Tadelis, "Consumer Heterogeneity and Paid Search Effectiveness" (2015). Gordon, Zettelmeyer, Bhargava and Chapsky, "A Comparison of Approaches to Advertising Measurement" (2019).