Robots Search Engine Crawler

Robots.txt file helps Search Engine to understand what need to be indexed and not to be. Rules written to this file are publicly visible and easily access by typing So take extra cautions when you are writing rule to this file.

  1. Download this empty file and start editing if you don’t have robots.txt file.
    [wpfilebase tag=file id=71 tpl=default /]
  2. Disallow everyone
    [wpfilebase tag=file id=20 tpl=default /]
  3. Disallow single robots
    [wpfilebase tag=file id=21 tpl=default /]
  4. Allow Everyone
    [wpfilebase tag=file id=22 tpl=default /]
  5. Allow Single Robot Spider
    [wpfilebase tag=file id=23 tpl=default /]
  6. Don’t allow directory
    [wpfilebase tag=file id=25 tpl=default /]
  7. Don’t allow single file
    [wpfilebase tag=file id=24 tpl=default /]
  8. Basic robots.txt file for WordPress site
    [wpfilebase tag=file id=26 tpl=default /]

Leave a Reply

Your email address will not be published. Required fields are marked *

Solve : *
4 × 2 =