Monograph IV · Historical data
Expired Domain Link Recovery Under US7346839B2
Google keeps a history of every URL it has ever scored. An expired domain is not a blank slate but a ledger, and the only way to recover its equity is to make the ledger balance.
| Record | US7346839B2 |
|---|---|
| Title | Information retrieval based on historical data |
| Inventors | Monika Henzinger, Jeffrey Dean et al. |
| Assignee | Google Inc. |
| Filed | 31 December 2003 |
| Granted | 18 March 2008 |
| Related schema | linkInfo history arrays, IndexingDocjoinerAnchorStatistics, redirectSourceUrl, soft-404 classifiers |
The Henzinger–Dean historical-data patent is remembered for its treatment of document age, but its longest section concerns links. The engine records, for every inbound link it discovers, when the link first appeared, when it was last seen, what anchor it carried and what the linking page looked like at the time. It records, for every URL, the set of such links over time, and it reasons about the rate of change of that set. A URL whose inbound links are growing steadily is treated differently from one whose links are static, and both are treated differently from one whose links suddenly point at content unlike anything they pointed at before.1US7346839B2, columns 9–13, describing link-based history data including “the appearance and disappearance of links,” “changes in anchor text,” and the use of those changes to detect “a document that has been taken over.” The claims explicitly contemplate expired domains re-registered for their link profile.
The ledger problem
Consider a domain that expires with 4,000 inbound links spread across 600 historical URLs: product pages, blog posts, a resources section, a contact page. Those links are entries in a ledger whose other side is the content each URL used to hold. When a new owner wires every one of those 600 paths to the root through a wildcard 301, the engine observes a single event repeated 600 times: a URL with a specific history now resolves to a page whose phrases, title, entities and outlink profile bear no relationship to that history. Under the patent, that is the signature of a takeover. Under the soft-404 classifier that runs on every redirect target, it is the signature of a catch-all page, which is exactly what it is. The links are not transferred. They are quarantined pending evidence.
equity_recovered(u) ≈ equity(u) · sim( content_now(target), content_then(u) ) · c(redirect_path)A working model. sim is the phrase-and-entity similarity between the redirect target and the URL’s historical content; c is a continuity factor that collapses when hundreds of unrelated URLs converge on one target.
Reconstruction, not redirection
The only approach that balances the ledger is to put the content back, or something the engine will accept as its continuation. The historical index itself, in the public web archives, supplies the phrase inventory of every URL that mattered. The reconstruction protocol is mechanical: rank the 600 historical URLs by inbound equity, rebuild the top tier at their original paths with content that carries the same good phrases and the same entity references, and allow the long tail to return a clean 410. A 410 is honest. The historical-data patent treats a link to a gone document as decayed, not as evidence of manipulation, and decayed equity does no harm.2In practice the top 10–15 percent of historical URLs typically hold more than 80 percent of the inbound equity. Rebuilding sixty pages well is more valuable than redirecting six hundred badly.
The zero-outdegree collector funnel
Rebuilt pages face a second problem: they must pass their recovered equity somewhere useful without repeating the convergence pattern that triggered the soft-404 inference. The solution that survives replication is a deep collector, a page two or three clicks below the root that sits on the same topic as the rebuilt cluster, receives one contextual link from each rebuilt page, and carries a single outbound link of its own. Its outdegree is one. By the PageRank division rule, the entire damped pool of the collector flows through that single link.
PR(collector) = (1−d)/N + d · Σ_i PR(rebuilt_i) / OBL_i
PR(destination) ← d · PR(collector) / 1Each rebuilt page keeps a modest outdegree, say three to six, so the collector receives a meaningful share from each. The collector’s outdegree of one passes 85 percent of its own rank onward with no division.
The pattern is invisible to the soft-404 and takeover heuristics because nothing converges: each rebuilt URL resolves to content consistent with its history, each links to the collector in the body with phrase-aligned anchors, and the collector is a normal topical page. What the engine sees is a site whose link graph was restored and then tidied. What the operator gets is a controlled, single-channel transfer of recovered equity to the destination of their choosing.
Order of operations
- Pull the historical URL inventory and inbound-link counts; rank by equity, not by page count.
- Reconstruct the top tier at original paths with phrase-faithful content; set 410 on the remainder.
- Restore the original internal navigation so
crawlPagerankrecovers and the rebuilt pages are re-scheduled. - Introduce the collector only after the rebuilt pages are re-indexed and holding position.
- Never redirect the collector. Link from it. Redirects are events; links are states, and the historical patent scores states.
Expired-domain work fails when it is treated as plumbing. It succeeds when it is treated as archival restoration with a link graph attached. The engine remembers what was there. Give it back.