Closed-Loop Ad Optimization
Most tools ship a change and never look back. This one logs a hypothesis with every optimization, then auto-grades it against real outcomes at day 3, 7, and 14 — and flags what isn't working. Genericized replica for Ridgeline Studio (Event venue + coworking · Google Ads).
Trace an optimization
Advance the clock and watch each change get graded against real metrics.
Increased Consumer Events daily budget $200 → $300
Click Day 3 to auto-grade the change against real post-change metrics.
Optimization actions
Increased Consumer Events daily budget $200 → $300
Paused keyword "cheap office space" — high spend, zero conversions in 30d
Paused "Corporate Holiday Parties" ad group (off-season)
Applied shared negative list (jobs, free, DIY) to Events (Local)
Raised bid on "coworking day pass" +25%
Launched "Birthday Party Venue" ad group
Paused legacy "Events (Local) — Broad" campaign
How it's built
Every Google Ads change is logged with a testable hypothesis and a baseline. A nightly job measures real outcomes at day 3/7/14 at the correct entity scope, grades each with a deterministic LLM pass (calibrated to the account's own volume, with a statistical floor so noise never reads as signal), and turns any "not working" verdict into a prioritized alert with a recommended revert. Production stack: Python + BigQuery + Postgres + Claude, on a nightly cron. This is a genericized, read-only replica.
the case study
The system behind this demo
Paid media optimization has a memory problem: changes get made, reasons get forgotten, and nobody checks whether last month's confident move actually worked. This system makes every change an experiment. A change enters as a hypothesis with its reasoning and baseline metrics. The nightly pipeline evaluates it at day 3, 7, and 14, at the level where its effect actually shows. And a deterministic verdict comes back — working, not working, or inconclusive — calibrated to this account's own volume, not industry averages.
in practice
How a change becomes a verdict
A change gets logged two ways. Deliberate optimizations are written into the ledger with a reason grounded in data, a testable expected outcome, and a snapshot of pre-change metrics. And the Google Ads change-event API is pulled nightly, so anything changed outside the ledger (a pause in the UI, a bid tweak) is captured anyway. Nothing escapes the record.
Then the system waits. At day 3, 7, and 14 the evaluator pulls before/after metrics at the right scope, hands them to a temperature-zero model along with the account's own volume baselines, and records a verdict with a two-sentence narrative: what happened to the key metric, and why the verdict is what it is. The daily report renders active experiments as a timeline, so "did that budget change work?" is a glance, not an archaeology project.
The practical outcome: the account's optimization history reads like a lab notebook. Real decisions came out of it — a budget tripled on the campaign the verdicts supported, spend halved on one they didn't, and a pause that looked like savings got flagged as a false economy when conversions fell faster than cost.
architecture
How it's put together
The hypothesis ledger
Every action is a JSONB record: action type, entity ids, what was done, the reason grounded in data, the expected outcome as a testable claim, and pre-change metrics. Changes without a hypothesis are just changes; this makes them experiments.
Nothing escapes the record
A nightly extractor queries the Google Ads change_event API for every change on the account — keyword pauses, bid moves, budget edits, new entities — so manual UI changes get captured and evaluated alongside the deliberate ones.
Scope routing
A keyword pause's effect is visible at the ad-group level; a budget change at the campaign level; a campaign pause is non-evaluable (there's nothing after to measure). Twenty-plus action types map to the scope where their effect actually shows, so the comparison is never apples to orchards.
Calibrated to the account, not the industry
Before evaluating, the system computes the account's own baselines — average daily clicks, median and 90th-percentile ad-group volume — and injects them into the prompt. Low-volume entities need a stronger signal over more days; high-volume entities can be read faster. No hardcoded thresholds.
Two evaluation paths
Standard changes get a delta evaluation: 14 days of pre-change baseline against the post-change window, with day-of-week patterns in view. New launches get a ramp evaluation instead, because a new ad group has zero history by definition and judging it against nothing is how tools lie to you.
Deterministic verdicts, validated loudly
Evaluations run at temperature zero, so the same inputs always produce the same verdict — re-runnable, auditable. Output is validated against the five allowed verdicts; anything malformed falls back to a template narrative and logs a loud warning instead of shipping garbage.
engineering highlights
The parts I'm proud of
A statistical floor the model can't talk past
The evaluator's prompt encodes the failure modes of eager analysis: entities below a minimum volume return insufficient_data no matter how dramatic the swing looks, and volume expectations scale to the account's own distribution.
UNIVERSAL STATISTICAL FLOOR:
- If entity has < 3 clicks/day in before-period, return "insufficient_data"
regardless of post signal. Sampling noise floor, independent of account size.
RELATIVE VOLUME CALIBRATION (using ACCOUNT BASELINE):
- Entity << median ad-group volume = "low for this account"
-> require stronger signal (30%+ shift across 5+ days)
- Entity >= p90 ad-group volume = "high volume"
-> standard eval rules apply; shorter trends readableHonest about launches
Most reporting hedges about new entities ("still ramping!"). This system forbids it: zero rows in the stats table means zero auction wins, and the prompt requires saying so.
ACTION_SCOPE = {
"keyword_pause": "ad_group", # effect visible at ad-group level
"ad_group_pause": "campaign", # effect visible at campaign level
"campaign_pause": "non_evaluable", # nothing after to measure
"budget_change": "campaign",
}
LAUNCH_ACTIONS = {"ad_group_create", "campaign_enable", "campaign_create"}
# LAUNCH TRUTH (prompt): zero rows = genuinely zero auction wins.
# Never frame as "pending", "not yet showing", "still loading".The verdict pipeline defends itself
LLM output is treated as untrusted input. Verdicts outside the allowed set get coerced to insufficient_data with a warning; narratives under length get replaced by a metric-driven template. Every evaluation stores its model, token usage, and cost, so the price of knowing whether changes worked is itself a tracked number.
verdict = result.get("verdict", "").strip().lower()
if verdict not in VALID_VERDICTS:
logger.warning(f"LLM returned invalid '{verdict}' — falling back")
verdict = "insufficient_data"
if len(narrative) < 20:
logger.warning(f"Short narrative len={len(narrative)} — using template")
narrative = _fallback_narrative(eval_day, verdict, before, after)Part of a larger reporting organism
stack
Built with
Python ETL (~10K lines, 4 extractors, 6 generators, 28 SQL transforms) · Google Ads change_event API via GAQL · BigQuery + GA4 sources · Supabase Postgres (raw/analysis/mart schema layers) · Claude (Haiku evaluations at temperature 0, Sonnet drafting, Opus grading) · Railway workers with per-run cost tracking.
The demo above replays real, genericized actions and verdicts. The thinking behind treating spend changes as experiments comes from operating budgets myself: the brand I ran and the model I ran it with.