But in Blogger the scenario is a little bit different. Here you will not maintain this robots.txt file. Rather google will do this for you :D. Dont be happy on this. In fact if google identifies duplicate content on your site, then google does this for its future crawl preferences. And you dont have the authority to modify your blogger's robots.txt. For details and for more details.
Monday, September 28, 2009
Restricted by robots.txt in blogger blog
I knew about the robots.txt. Suppose you have a lot of pages in your site. Among them you dont want to expose all of your pages to search engines. To do this for google the way is robots.txt. You should list the URLs in the robots.txt file. Google honors this file ad dont crawl those links.
Subscribe to:
Post Comments (Atom)
At least you can see you robots.txt on this link.
ReplyDeletehttp://your_blog_address.blogspot.com/robots.txt
OR
you can also see this on the google webmaster tool. Follow this menu navigation
Site Configuration -> Crawler Access