The short answer
Duplicate content means several URLs return the same or nearly the same page. There is no penalty for it: Google says internal duplication is a technical situation, not a violation. The actual problem is that the search engine then decides on its own which address to show — and it does not always pick the one you have been building links and content for.
On a normal site the overwhelming majority of duplicates come not from copied text but from the way the CMS generates addresses: a trailing slash, www, http, UTM tags, sorting parameters, a product sitting in two categories. Three tools fix this — a 301 redirect, a canonical tag and noindex — and which one you pick depends on a single question: does a real visitor need that second URL?
Where duplicates actually come from
When a client says "we have duplicate content", they usually mean copied text. Working with Estonian sites, I run into copied product descriptions far less often than into technical duplicates. Here are the sources in the order of how frequently I see them.
| Source of duplicates | Example | Usual cause |
|---|---|---|
| Variants of one address | /services and /services/, www and non-www, http and https | No single URL format enforced at server level |
| GET parameters | ?utm_source=, ?sort=price, ?sessionid= | Ad tags and internal catalogue features |
| Product in several categories | /lamps/desk-lamp and /for-office/desk-lamp | CMS builds the URL from the category path |
| Filters and faceted navigation | ?color=black&size=xl in any combination | The catalogue returns 200 for any parameter combination |
| Technical copies | print view, AMP leftovers, /index.php | Legacy of old templates and migrations |
| CMS auto-generated pages | date, tag and author archives, WordPress attachment pages | On by default, never switched off |
| Identical text across pages | templated location or city pages | Landing pages mass-produced without unique content |
The important consequence: duplicates almost always arrive in classes, not one by one. If you found one product reachable at two addresses, the entire catalogue is reachable at two addresses. That is why, when I find a duplicate, the first thing I look for is the rule that created it rather than the individual page.
It is also worth separating duplicates from keyword cannibalization. A duplicate is the same content at different addresses. Cannibalization is different pages with different content competing for the same query. The first is fixed by consolidation, the second by reworking structure and content, and confusing the two costs weeks.
What duplicates actually cost you
Three effects, of very different size.
Google shows the wrong URL. This is the big one. The search engine picks the canonical address itself, weighing links, sitemap entries, canonical tags and internal linking. When those signals contradict each other, a URL with a UTM tag or a product from a secondary category can end up in the results. Traffic looks fine on the surface, but the accumulated signals are spread across addresses.
Crawl budget is wasted. On a site of a few thousand pages this is usually not a concern — Googlebot will get through everything. On a catalogue where filters generate tens of thousands of combinations it changes: the crawler spends its visits on junk addresses while new product pages wait weeks for indexing. The symptoms overlap with what I described in the article on why Google is not indexing your pages.
Internal link equity is split. If some internal links point to /services and others to /services/, the equity is divided between two addresses instead of one. A smaller effect than the first two, but it disappears for free once the linking is cleaned up.
What duplicates do not do is demote the whole site. If someone tells you your site is "filtered for duplicate content", ask to see the specific report. In the vast majority of cases that phrase covers an ordinary ranking drop with a different cause.
How to find duplicates: three sources
No single source gives the full picture, so I always check all three.
1. Search Console, the Page Indexing report. The most honest source: it shows not what you think is a duplicate but what Google treats as one. The statuses that matter:
- "Duplicate without user-selected canonical" — Google found identical content and consolidated the addresses itself because you did not set a canonical.
- "Duplicate, Google chose different canonical than user" — you set a canonical and Google disagreed. That is a signal either that the pages really are different, or that your internal links and sitemap contradict your canonical.
- "Page with redirect" in large numbers — often a trace of redirects working correctly while internal links still point at the old addresses.
2. A crawl of the site. Run Screaming Frog, Sitebulb or any other crawler and sort the result by three fields: title, H1 and content hash. Matching titles are the fastest indicator of a duplicate class. Check separately whether the site returns 200 for an address with an arbitrary parameter appended: if /services?foo=bar opens as a normal page, you have a potentially infinite number of duplicates.
3. The search results themselves. A query like site:yourdomain.ee "a snippet of text in quotes" shows how many addresses Google keeps in the index with that text. A blunt instrument, but it catches what never made it into the reports: old subdomains, staging copies, leftovers from an unfinished migration.
A fourth source people forget is sitemap.xml. If your sitemap lists parameterised or redirecting addresses, you are telling Google with your own hands that those URLs are canonical. How it should be set up is covered in my article on sitemap.xml and robots.txt.
What to consolidate with: 301, canonical or noindex
One question decides the tool: does a real visitor need the second URL?
| Situation | Tool | Why |
|---|---|---|
| Old address after a structure change | 301 redirect | URL is no longer needed, signals pass on, address leaves search |
| Trailing slash, www, http | 301 at server level | One canonical format for the whole site |
| UTM tags and sorting parameters | canonical to the clean URL | The link must work, but it should not be indexed |
| Product in two categories | canonical to the primary path | Both pages are needed in navigation |
| Pagination | self-referencing canonical | Pages 2+ list different products and are not duplicates |
| Print view, PDF copy | canonical to the HTML version | Useful to people, not to search |
| Cart, account, internal search | noindex | Needed by visitors, useless in results |
| Genuinely worthless auto-generated archives | noindex, then removal | No value to anyone |
Three things I regularly see done wrong:
- A canonical is a hint, not a directive. Google weighs it against other signals and can ignore it. If you set a canonical to page A while every internal link and the sitemap point to B, B wins. The signals have to agree.
- Do not block duplicates in robots.txt. A page disallowed in robots.txt is never fetched, so the crawler never sees the canonical or the noindex on it. The address stays in the index as "indexed, though blocked by robots.txt" and no consolidation happens. Robots.txt is for saving crawl effort, not for removing pages from the index.
- Do not combine noindex and canonical on the same page. That is a contradictory instruction: the canonical says "pass signals over there", noindex says "drop me". Pick one.
Language versions are not duplicates
On the Estonian market this is the most expensive mistake on the list, because nearly every site here is multilingual.
The Estonian, Russian and English versions of a page contain different text and address different audiences — by definition they are not duplicates, even when images, prices and layout match. Worse: a canonical from the Estonian version to the Russian one removes the Estonian page from Estonian-language results entirely, handing that local traffic to a competitor.
The correct setup is simple: every language version has a self-referencing canonical, and the versions are tied together with hreflang, including a self-reference. The typical mistakes in that markup and how to verify it are in my separate article on hreflang setup.
A separate case is one language across several domains or subdomains — for example .ee and .com carrying the same English text. Those are genuine cross-site duplicates, and the answer is either a cross-domain canonical or an honest split of the content.
What I see in practice
A few observations from Estonian projects that repeat again and again.
The most common duplicate is not in the catalogue but on the home page. The site opens at four addresses at once: with and without www, over https and http. This is configured once at server or CDN level and closes an entire class of problems before anyone even looks at product pages.
Second most common are parameters from internal features. Sorting, items per page, session identifiers, traces of an old filter. Each of them doubles or triples the number of catalogue addresses.
After a migration duplicates appear even when the redirects are correct. The redirects get done, but internal links in the menu, in article text and in the sitemap keep pointing at the old addresses. Formally everything works; in practice Googlebot walks redirect chains and receives contradictory signals. The check is simple: after a move, a crawl should not find a single internal link to an address that returns 301.
Duplicates often sit next to orphan pages. Both come from the same root cause — the real structure of the site and what the CMS believes about it have drifted apart. While you are sorting out one class, it makes sense to check the other; how to find them is in the article on orphan pages.
An honest limit: everything above applies to sites up to a few tens of thousands of URLs, which is what I work with most. On large marketplaces, parameter handling becomes an engineering problem of its own, and the decisions are made at the level of catalogue architecture rather than CMS settings.
The order of work
- Check whether the site opens at several variants of the home address and set up a 301 to one canonical format.
- Open the Page Indexing report in Search Console and list the pages with duplicate and foreign-canonical statuses.
- Run a crawl and group pages by matching titles and H1s — that reveals classes rather than isolated cases.
- For each class, decide whether a visitor needs the second URL and pick the tool from the table above.
- Bring internal links and sitemap.xml in line with the canonical addresses you chose — that is half the effect.
- Verify that language versions are connected with hreflang rather than consolidated with canonical.
- Come back to the Page Indexing report in three to four weeks and check whether Google agreed with your choice.
That last step is skipped most often, and it is the actual verification: if Google still chooses a different canonical, a contradicting signal is still in place somewhere. If the right addresses need to be built into the architecture of a new site instead, that is a web development task rather than a set of point fixes.
