Blogger Adds Robots.txt

If you use Google Sitemaps for your Blogger and you are seeing a sudden increase in the number of links being blocks by robots.txt, don't panic as it is not anything you did wrong. It is all because Blogger is adding robots.txt by default recently. As you can see from the codes below, all pages under the /search directory is being disallowed meaning and pages under the /search directory would not appear included in any search result pages of major search engines (e.g. Google, Yahoo and Live.com).

User-agent: *
Disallow: /search
Sitemap: http://gspy.blogspot.com/feeds/posts/default?orderby=updated

This is, in fact, a good news to Blogger users as Google is treating most of these blocked pages as duplicated content and are listing them as 'Supplemental Results'. Furthermore, the more duplicated content you site has, the less Google is weighing your site's content.

A further improvement towards this great feature is to allow Blogger users to customize their own robots.txt so they could prevent undesired content from appearing in search results.

Note: If you are using Blogger and has recently directed your feed to FeedBurner, make sure you change your sitemap URL in Google Sitemaps to http://yourblog.blogspot.com/rss.xml?orderby=updated instead of just http://yourblog.blogspot.com/rss.xml otherwise an error would occur in Google Sitemaps.

Know more about links to you

Google updated its webmaster tool which now shows external and internal links for a particular page. Before, webmaster will have to use the 'link:' attribute in Google search to check backlinks (in Google's language, it is called external links). You get much more up-to-date and detailed linkage from this tool and I have to admit I like it very much. If you got a large site and got loads of links in and out, you can export the data to Excel. As mentioned in the Official Google Webmaster Central Blog:

... the overall fraction of links we show is much, much larger than the link: command currently offers.

So you might get excited (like me) who saw the astonishing amount of links to your site. Another small update is the 'My sites' drop down menu in the top right corner of the page which allows much easier navigation between sites you own. If you own a site but haven't tried this out yet, I strongly recommend you give it try.

Reporting Spam in Google

Many webmasters are aware that Google provides an easy-to-use sitemap tool which allows them to upload supported formats of sitemap and notify Google of changes of their site, but not many of them have notice that you can actually help improve Google's search results by reporting spam sites. On the front page of the sitemap tool, you will see the Tools link, on click, the Report spam in our index and you will be brought to a submission form. These days, there have been increasing complains about the amount of spam sites Google includes in its search service especially for Blog Search. And as the founder of Wikipedia has expressed interest in competing with Google by setting up a search engine (Wikiseek) which restricts its searches to pages from Wikipedia and those sites that are referenced within Wikipedia aiming to reduce spam.