Blogger SEO: How to setup robots.txt for blogger | Crawlers, Indexing

What is robots.txt?

• The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.
• By default search engine bots crawl everything possible unless they are forbidden from doing so. They always scan the robots.txt file before crawling the web site.
• Declaring a robots.txt means that visitors (bots) are not allowed to index sensitive data but it doesn’t mean that they can’t. The legal/good bots follow what is instructed to them but the Malware robots don’t care about it, so don’t try to use it as a security for your web site.

Blogger SEO: How to setup robots.txt for blogger | Crawlers, Indexing

WHERE TO PLACE ROBOTS.TXT:

1. login to blogger.com

2. goto SETTINGS

3. click on  SEARCH PREFERENCES

4. place below code in the showed text area

User-agent: Mediapartners-Google
Disallow: 
User-agent: *
Disallow: /search?q=*
Disallow: /*?updated-max=*
Allow: /

Sitemap: http://www.aitamelearning.blogspot.com/feeds/posts/default?orderby=updated

 change the redcoloured link with another

ENJOY!!!!!