Keyword cannibalization is not a spreadsheet problem. Most teams hunt for it by exporting queries, filtering for two URLs that rank for the same term, and arguing about it in a meeting. That method only catches pages that already collide in Google. It misses the quieter version, where five posts say the same thing in different words and none of them ever gets enough authority to rank.
Embeddings fix this. Convert every page on your site into a vector, measure how close the vectors sit, and you get a ranked list of pages competing for the same meaning. It takes an afternoon, costs a few cents, and finds overlap that rank trackers cannot see.
Why Query Overlap Misses Most Cannibalization
The standard check looks at Search Console data: one query, multiple URLs. That is useful, but it is a lagging signal. A page has to be indexed, ranking, and getting impressions before it shows up, and by then you have already wasted months of crawl budget and link equity.
Semantic overlap is the leading signal. Two articles can target different keywords and still answer the same question for the same reader. Search engines, and the AI answer engines that increasingly sit on top of them, cluster content by meaning, not by string match. If your site has three pages that mean the same thing, the system picks one and ignores the rest, or picks none.
| Method | Catches | Misses | Effort |
|---|---|---|---|
| Manual query export | Pages already ranking for the same query | Pages that overlap in meaning but not wording | High |
| Title and H1 similarity | Near-identical headlines | Different headlines, same argument | Low |
| Embedding similarity | Overlap in meaning across the whole site | Intent differences a human must judge | Low, then one review pass |
Build the Similarity Map in One Script
The pipeline has four steps. Pull every indexable URL from your sitemap. Extract the title, H1, and the first 500 words of body copy, because that is where a page declares what it is about. Send that text to an embedding model in batches. Then compute cosine similarity between every pair of pages.
Keep the code boring. A few dozen lines of Python with numpy handles a site of several thousand pages, since the pairwise matrix for 2,000 pages is only four million comparisons. Store the vectors in a file, not a vector database. You are running this monthly, not serving it in production.
Two details matter more than the model you pick. First, embed the same fields for every page, or the template text will dominate the vectors and every page will look alike. Strip navigation, footers, and boilerplate before you embed. Second, record the model name and version next to the vectors. If you switch models later, you have to re-embed everything, and scores from different models are not comparable.
Turn Scores Into Decisions
A similarity score is not a verdict. Set three bands and attach an action to each one.
Above roughly 0.90, treat the pair as a likely duplicate. Pick the stronger page by backlinks and traffic, merge the weaker one into it, and 301 redirect the old URL. Between 0.80 and 0.90, the pages overlap but may serve different intents. Open both, compare the search results each one targets, and either sharpen the angles or consolidate. Below 0.80, leave them alone and consider an internal link between them instead.
Treat those thresholds as starting points. Every model produces a different score distribution, so calibrate on 20 pairs you already know are duplicates and 20 you know are distinct. Pick the cutoffs that separate them. Ten minutes of labeling beats trusting a number from a blog post, including this one.
Then join the results to Search Console. A high-similarity pair where both pages earn impressions for the same queries is your highest priority fix. A high-similarity pair where neither page ranks is still worth merging, because the combined page has a better chance than either alone.
Make It a Monthly Gate, Not a One-Time Audit
The real value shows up when this runs before you publish. Embed the draft, compare it to the existing library, and block publication if anything scores above your duplicate threshold. The writer sees the closest existing pages and decides whether to update one instead of shipping another near-copy. That one rule prevents most cannibalization from ever being created.
- Export indexable URLs from the sitemap and drop paginated and parameter pages
- Embed title, H1, and the first 500 words, with boilerplate stripped
- Log the embedding model name and version alongside the vectors
- Calibrate duplicate and overlap thresholds on 40 hand-labeled pairs
- Join flagged pairs to Search Console impressions and rank the fix list
- Add a pre-publish check that compares every new draft to the library
Run the first pass this week and expect to find more overlap than anyone on the team believes exists. Merge the worst offenders, redirect cleanly, and let the survivors collect the authority that was being split. Cannibalization is a measurement problem, and embeddings give you the measurement.
Tags
LETSGROW Dev Team
Marketing Technology Experts
Ready to Apply This Insight?
Schedule a strategy call to map these ideas to your architecture, data, and operating model.
Schedule Strategy Call