Robots Search Engine Crawler Control.

Robots Search Engine Crawler

Robots.txt file helps Search Engine to understand what need to be indexed and not to be. Rules written to this file are publicly visible and easily access by typing http://yourdomain.com/robots.txt. So take extra cautions when you are writing rule to this file.

  1. Download this empty file and start editing if you don’t have robots.txt file.
    [wpfilebase tag=file id=71 tpl=default /]
  2. Disallow everyone
    [wpfilebase tag=file id=20 tpl=default /]
  3. Disallow single robots
    [wpfilebase tag=file id=21 tpl=default /]
  4. Allow Everyone
    [wpfilebase tag=file id=22 tpl=default /]
  5. Allow Single Robot Spider
    [wpfilebase tag=file id=23 tpl=default /]
  6. Don’t allow directory
    [wpfilebase tag=file id=25 tpl=default /]
  7. Don’t allow single file
    [wpfilebase tag=file id=24 tpl=default /]
  8. Basic robots.txt file for WordPress site
    [wpfilebase tag=file id=26 tpl=default /]

Published by

Emdad Khan

I am Emdad Khan, A WordPress Theme Developer and Author of BN Web Design Website. A full time Freelancer on WordPress Theme Development & Optimization as well as blogger.

Leave a Reply