Instructing the Robots
It is useful to be able to give directions or hints to search engines as there are often parts of your website that do not need to be indexed, for example javascript files. The two main ways to give instructions are:
- the Robots.txt file
- the Robot meta tag
Robots.txt
This is a plain text file that tells search engine what not to bother with on your site. For example a folder that contains javascript, or one that you use to hold some development work in which isn't ready for public consumption. You place the file (called robots.txt) in your websites home directory.a typical file will look like this:
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/
Robots Meta Tag
Use the Robot Meta tag to instruct the search engine what to do with a page. You can instruct them not to index the page and not to follow the links on a page.