Almost 2 months ago Google made a small update to their search index. Small to most people, but large to me as Sudoku.com.au dropped from #3 for the keyword ’sudoku’ to #21. Not across all datacenters though, on google.com.au I was still #1.
I checked out what had happened to other sites, and noticed that a few .co.uk sites had also dropped out out of the top 10 into the 20s - while these same sites where still in the top 5 in google.co.uk. So my first thought was that there is some kind of localisation going on; Sites that end in .com.au get a bonus at google.com.au, but are penalised at google.com. This definitely occurs already, but it was surprising to have such a big jump in rankings.
Then I came across this attribute you can add to google search urls, which show ‘unfiltered’ search results. Just add:
to the end of the google url.
Remove the filter, and I’m back at #4!
So what is being filtered?
I checked around, and it seems that adding &filter=0 mostly removes the ‘duplicate content’ filter. So if your site has duplicate content - either with another site, or within the site itself - it can have a penalty.
This immediately led me to do the site: command, which lists all the pages of your site that google knows about
It came back with 4 results! - the rest were left out because google thought they were similar to the 4 already shown. Sudoku.com.au has about 25,000 pages, all with unique comments by visitors. Something was up, and all of a sudden my localisation theory didn’t look so strong.
So what to do?