Compare cited sources over time

What needs to remain comparable?

Start with a named question panel and preserve its exact wording. Record the model or interface, language, region, search mode, collection dates and number of completed answers. A new model version or expanded topic set can change the observed source list even when the publisher’s content has not changed.

Use the same source extraction and URL normalization rules in each period. Keep original article URLs alongside normalized URLs and domains. A familiar domain may cite a completely different article next month, so domain retention alone can conceal a change in evidence.

The public sampling-completeness preprint shows why observed source accumulation depends on the number of collected runs. Compare equivalent coverage or label the mismatch; do not call a longer collection a source gain without accounting for the additional opportunity to observe links.

Which calculations are useful?

Let the earlier domain set be A and the later set B. Report retained domains in both sets, newly observed domains in B only and domains observed earlier but not in B. Define the denominator for any percentage.

An illustrative earlier set contains six domains, and the later set contains seven, with four in common. Retention relative to the earlier set is 4/6. Overlap relative to their union is 4/9. Both calculations are valid descriptive summaries, but they express different things. Neither measures how much a model trusts the outlet.

Include citation frequency as a separate field. A domain present in both periods may fall from many observations to one. Conversely, a single newly observed link should not dominate the report simply because it is new.

How should you investigate a change?

Read a sample of the retained and changed citations. Check whether a new story, changed page, altered query interpretation or temporary retrieval issue provides a plausible explanation. Record it as a hypothesis unless the evidence establishes more.

Use citation-verifiability principles to inspect support at the claim level. Repeated citation of an irrelevant or outdated page is not a success merely because it is stable. Preserve no-answer and failed-run cases so missing observations are not mistaken for source disappearance.

In Rankfor Index, Source & Language and Source history provide distinct views of the fixed instrument’s source evidence. Keep that instrument separate from a custom Answer Trail panel. Record which measurement produced each table and compare equivalent source lists within their own series.

What does this mean for a publishing decision?

Repeated observed citation can justify investigating an outlet’s relevance, editorial quality and commercial terms. It does not make the outlet “safe to fund” or guarantee that a placement will change answers. Review readership fit, factual standards, disclosure and rights independently. The budget decision should state what the source history contributes, what it cannot establish and what evidence would justify continuing the investment.