HOW TO USE ROBOTS DOT TXT FILE FOR SEARCH ENGINE

Robots dot txt files plays an important role in search engine result page (SERP). It gives direction to search engine crawlers what to index and not. This file must be placed in root directory of website and easily accessible by crawlers.

HOW TO CREATE ROBOTS DOT TXT FILE?

Creating robots.txt file is very easy. Just create a new text file and name it robots.txt.

HOW TO USE ROBOTS DOT TXT FILE?

To use robots.txt file you must add some rules for search engine crawlers. Here are some examples

ROBOTS DOT TXT FILE: DISALLOW EVERYONE TO ACCESS WEBSITE

User-agent: *

Disallow: /

EXPLANATION: robots.txt file giving instruction to all search engine crawlers not to index website.

ROBOTS DOT TXT FILE: DISALLOW SINGLE USER TO ACCESS WEBSITE

User-agent: BadBot

Disallow: /

EXPLANATION: Robots dot txt file giving permission to access everyone except BADBOT user agent.

ROBOTS DOT TXT FILE: ALLOW EVERYONE

User-agent: *

Disallow:

EXPLANATION: robots dot txt file giving access to index website for every search engine crawler.

ROBOTS DOT TXT FILE: ALLOW SINGLE USER

User-agent: Google

Disallow:

EXPLANATION: Robots dot txt file giving permission to Google to access and index website.

ROBOTS DOT TXT FILE: DON’T ALLOW DIRECTORY

User-agent: *

Disallow:  /folder1/

Disallow:  /folder2/

Disallow:  /folder3/

Disallow:  /folder4/

EXPLANATION: Robots dot txt file telling everyone not to access and index folder1,2,3,4.

ROBOTS DOT TXT FILE: DON’T ALLOW SINGLE FILE

User-agent: *

Disallow:  /folder1/example.html

EXPLANATION: Robots dot txt file telling everyone not access and index example.html file inside folder 1.

For WordPress user, here is ready to go robots.txt file. Copy, paste and upload to WordPress root directory.

User-agent: *

Disallow: /wp-admin/

Disallow: /wp-includes/

Disallow: /?s=/

Disallow: /?attachment_id=/

VIDEO ON HOW TO USE ROBOTS DOT TXT FILE FOR SEARCH ENGINE

Leave a Reply

Your email address will not be published. Required fields are marked *

Solve : *
18 − 4 =