Not several net master take the time to use a robots.txt file for his or her web site. For search engine spiders that use the robots.txt to visualize what directories to go looking through, the robots.txt file will be terribly useful keep the spiders indexing your actual pages and not different data, like exploring through your stats!
The robots.txt file is helpful keep your spiders from accessing components folders and files in your hosting directory that are totally unrelated to your actual net web site content. you'll opt to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to stay them out of the online stats portion of your web site.
Many search engines cannot read dynamically generated content properly, mainly created by programming languages, like PHP or ASP. If you have got an internet store programmed in your hosting account, and it's during a seperate directory, you'd be wise to dam out the spiders from this directory thus it solely finds relevant data.
The robots.txt file ought to be placed within the directory where your main files for your hosting are located. thus you'd be suggested to form a blank text file, and put it aside as robots.txt, and then upload it to your net hosting to identical directory your index.htm file is found.
Here is samples of the utilization of the robots.txt file:
To block out a directory during a robots.txt file, like a subdirectory for your on-line store known as /store/ you'd do the following: Disallow: /store/
Another example to dam out your stats directory: Disallow: /stats/
You may additionally need to disallow individual files that you simply don't need searched by the search engines. as an example you dont need search.php to be parsed by the Search Engines. to try to to this you sort within the following on its own line:
Disallow: /search.php
Following the principles made public and making the robots.txt file, you may keep search engine spiders out of unwanted files and directories, and letting them undergo the necessary files to visualize what your net web site is all about!
The robots.txt file is helpful keep your spiders from accessing components folders and files in your hosting directory that are totally unrelated to your actual net web site content. you'll opt to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to stay them out of the online stats portion of your web site.
Many search engines cannot read dynamically generated content properly, mainly created by programming languages, like PHP or ASP. If you have got an internet store programmed in your hosting account, and it's during a seperate directory, you'd be wise to dam out the spiders from this directory thus it solely finds relevant data.
The robots.txt file ought to be placed within the directory where your main files for your hosting are located. thus you'd be suggested to form a blank text file, and put it aside as robots.txt, and then upload it to your net hosting to identical directory your index.htm file is found.
Here is samples of the utilization of the robots.txt file:
To block out a directory during a robots.txt file, like a subdirectory for your on-line store known as /store/ you'd do the following: Disallow: /store/
Another example to dam out your stats directory: Disallow: /stats/
You may additionally need to disallow individual files that you simply don't need searched by the search engines. as an example you dont need search.php to be parsed by the Search Engines. to try to to this you sort within the following on its own line:
Disallow: /search.php
Following the principles made public and making the robots.txt file, you may keep search engine spiders out of unwanted files and directories, and letting them undergo the necessary files to visualize what your net web site is all about!
Posted in: Search Engine Optimization
0 comments:
Post a Comment