Do you need a Robots.txt file? When you have a small site, you are probably under the false assumption that you really don't nemased a robots.txt file. In fact, you may be saying to yourself, "I don't need a robots.txt file because, my site is, small, it's simple for the search engines to find, and since I want all pages indexed anyway, why bother." That was my thoughts in the beginning, as well as, not being aware of what a robots.txt file is/was or what it could do for my site. Thus, I'll try to give you a little insight as to what a robots.txt is, how to use them, why you need them and some basic instructions on creating a robots.txt file.
Define Robot.txt File
To begin we need to know what a web robot is, and is not. Thus, a Web robot is sometimes called spiders or web crawlers. These should not be confused with your normal web browser, for a web browser is not a web robot because a human being manually maneuvers it.
The main use of a robots.txt file is to give robots instructions to what they can crawl and what they should not crawl. This gives you a little more control over the robots. And since this gives you a little more control over the robots, which means you can issue indexing instructions to specific search engines.
Posts
