Adding custom Robots.txt Setting in blogger

Welcome guys..our today topic is custom Robots.txt Setting in blogger. In my previous post i have discussed about Custom Robots Header Tags Setting in Blogger. if you have read that post then i hope you guys are aware with it's importance in Google page ranking. if not then please read it. Adding Custom Robots Header Tags Setting in Blogger really important to your blog. now our next topic is Adding custom Robots.txt Setting in blogger. Custom Robots.txt is a way for you to instruct the search engines bots like Googlebot(Web Crawling Robot of Google) that you don't want it to crawl certain pages of your blog. In this tutorial i will tell you how to add custom Robots.txt Setting in blogger.

Robots.txt

Robots.txt is a text file and saved on the websites and blog's servers which instruct the search engines bots like Googlebot that you don't want it to crawl certain pages of your blog. Search Crawlers always scan the Robots.txt file first and then other web pages.

Each blog hosted on blogger have it's default Robots.txt file shown below

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED

Information about each line

Code is divided into three sections shown below

1. User-agent: Mediapartners-Google
 This code is for Google Adsense robots which help them to serve better ads on your blog.

2. User-agent: *
User-agent is for all robots marked with asterisk (*). In default settings our blog's labels links are restricted to indexed by search crawlers that means the web crawlers will not index our labels page links because of below code.

Disallow: /search

Allow: / refers to the main page that means web crawlers can crawl and index our blog's main page. 

3. Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED
This refers to the sitemap of our blog. By adding sitemap link here we are simply optimizing our blog's crawling rate.

Above sitemap will work for recent 25 post.if you want to increase no of links in your sitemap then replace default sitemap section with below code:

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

And if you have more than 500 published post then use below code

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000 

Adding custom Robots.txt Setting in blogger
There are some steps to add Robots.txt in blogger shown below:

Step 1: Please log in to your Blogger account.

Step 2: Then click on settings  >  Search preference

Step 3: Click on Edit link under the Custom Robots.txt  section which is shown below image

Step 4: Then click on Yes option.

Step 5: Now paste your robots.txt file code in the box.which is shown below image

Step 4: Then click on Save button.You are done!


Check your Robots.txt File
You can check this file on your blog by adding /robots.txt at last to your blog URL in the browser.

Example: http://teck-deal.blogspot.in/robots.txt

See also: SEO tips for blogspot blog