Task Stability: we tested whether our own measurement holds

Why this test exists

A visibility score compares how a brand does across AI engines and across prompts. That comparison rests on an assumption most tools never check: that a given prompt asks the engine to do the same job every time you run it. If the executed task drifts from run to run, then a single scan is measuring a moving target, and any number built on it is unstable at the root.

When we described our approach in r/GEO_optimization, a practitioner raised exactly that objection: the run count was too low to trust, because the task itself might vary between runs. It was a fair objection, and the honest response to a fair objection is to measure it rather than argue with it. So we pre-registered a protocol on 25 August 2026 and ran it.

What we measured

120 runs. Three prompts, two engines (ChatGPT and Perplexity), twenty runs each, one account, one location, one week, on the consumer surface.

The three prompts were chosen to span three classes:

• Transactional: "Best AI visibility tracking tool for a B2B SaaS company"
• Explanatory: "What is generative engine optimization"
• Ambiguous: "How do I know if AI is recommending my brand"

Every answer was labeled by executed task using a taxonomy proposed by a practitioner in r/GEO_optimization (handle Upstairs_Control_611): Recommendation, Explanation, Comparison, Troubleshooting, and a deliberate residual bin, Other or unclear. A secondary field recorded the answer frame, such as whether a recommendation arrived as a ranked list or as a single option advanced. Labeling was done on a blind pass that stripped engine, prompt, and run order.

We pre-registered the decision rule before seeing any data. A cell counts as Stable only if the lower bound of its exact binomial interval clears 80 percent. That bar has a consequence worth stating plainly: at twenty runs, a perfect 20 of 20 is the only outcome that clears it. 19 of 20 lands at 75.1 to 99.9 percent and would have been called Inconclusive. So the design could return "Stable" only on a clean sweep. It was built to make stability hard to claim, not easy.

The headline result, and it is a null

Every one of the six cells returned 20 of 20 on a single executed task.

Every cell cleared the pre-registered 80 percent lower bound. The executed task did not vary across runs in any cell. The objection that prompted the test was wrong at this scope, and we will say so in those words: task was stable everywhere, and a single run was enough to observe it. That is the null we pre-registered, and we are reporting it as it came back.

What turned out to be more interesting than the null

The test asked whether task varies across runs. It does not. But the same data answered a question nobody asked, and the answer is louder than the headline.

Task varies across engines, deterministically, on two of the three prompts

Same prompt, same week, same conditions, different executed task on every single run:

• On the explanatory prompt, ChatGPT executed Explanation twenty times out of twenty. Perplexity executed something the taxonomy could not name, twenty out of twenty.
• On the ambiguous prompt, ChatGPT executed Explanation twenty out of twenty. Perplexity executed Troubleshooting twenty out of twenty. Not a difference in emphasis. A different job.
• On the transactional prompt, both executed Recommendation. The one prompt of the three where the engines agree on what was asked of them is the unambiguous commercial one.

The executed task is a stable property of the prompt and engine together, not of the prompt alone. This helps and it hurts. It helps because determinism means one run measures it, so the check is even cheaper than we assumed. It hurts because a cross-engine comparison on a prompt like the ambiguous one is not two engines answering one question with different success. It is two engines answering two different questions. On that prompt ChatGPT is explaining a concept and Perplexity is diagnosing a failure state. Any comparative score computed across those two answers is comparing an explanation to a diagnosis, and nothing in the output signals it.

So the precondition for a cross-engine comparison is not "is the task stable across runs." It is "do these two engines execute the same task on this prompt at all." Check it once. Then compare, or do not.

A stable cell was hiding real variation one level down

On the transactional prompt, ChatGPT executed Recommendation 20 out of 20, cleanly Stable. But the answer frame split 15 single option advanced and 5 ranked. Fifteen of twenty is 50.9 to 91.3 percent, which is Inconclusive on frame by the same rule that called the task Stable. Perplexity, on the same prompt, produced a ranked list all twenty times with no split at all.

This is the most useful methodological finding in the run. The task label was coarse enough to report perfect stability while a real difference moved underneath it. Whether a recommendation arrives as a ranked table or as a single option advanced is not cosmetic. It changes whether "position" is even a defined quantity in the answer, and one engine produces a ranked list every time while the other produces one a quarter of the time. That is the cell to extend to thirty runs, which the protocol already said to do for an interesting and inconclusive cell rather than for all of them.

The taxonomy failed on one engine, and that is a finding about the taxonomy

Perplexity's explanatory answers landed in Other or unclear on all twenty runs. Twenty out of twenty in a catch-all bin does not mean the bin is doing its job. It means the bin the answers belonged in does not exist. Perplexity consistently defined the concept and contrasted it with a neighboring one, which tripped the exclusion in the Explanation rule, but the two things it contrasted were methodologies, not options a buyer chooses between, so it did not fit Comparison either. Other or unclear was the correct call under the rules as written, which is exactly why keeping a residual bin was worth it. The fix belongs to the taxonomy, either by scoping "options" to purchase options or by adding an explanatory-with-contrast label. That is a contribution back to the label set, and it was only findable because the residual bin existed and was not optional.

Limitations, stated before anyone else states them

• One account, one location, one week, on the consumer surface, not the API.
• Three prompts. Each prompt-class conclusion rests on a single prompt and should be read as illustrative of the class, not as a measurement of it.
• Hand labeling by an interested party. The blind pass stripped engine, prompt, and run order, but it did not strip writing style, and these two engines are distinguishable by voice. That reduced the risk of a biased call. It did not eliminate it.
• Twenty runs can demonstrate instability and can barely demonstrate stability. Six cells returned the only outcome that clears the bar. The design would have called a genuinely 95-percent-stable cell Inconclusive most of the time.
• Search was triggered on the recorded runs, so retrieval variation between runs is inside the result rather than controlled out of it.

What it changes in the product

This test changed the precondition we hold a cross-engine comparison to. A comparison across two engines on a prompt should first check, once, whether the two engines execute the same task on that prompt. If they do, the comparison stands. If they do not, it should be labeled not comparable rather than reported as a number that looks like a measurement and is not.

We are building that check into the product. It is not live yet, and we are not going to describe it as if it were, because that would be the exact move this page argues against. When it ships, this section will say so and will say when. The rule behind it is the one we apply everywhere else: a number we cannot stand behind should say so instead of being shown.

Credit

The task taxonomy is the label set proposed by Upstairs_Control_611 in r/GEO_optimization. The protocol was pre-registered in that thread on 25 August 2026, before any data existed, including the decision to keep a residual bin, which is what turned the taxonomy's one failure into a finding instead of a gap.