|
Dictionary > Definitions > Search Engines > robots.txt  robots.txt
DefinitionFile of directives intended for the robots of the search engines. DescriptionOf format text and placed at the root of a site, this file makes it possible to prohibit with the robots search engines the access to certain repertories or certain files.
It is possible to give directives specific to certain robots.
Only the closed areas are indicated. All the remainder is regarded as being authorized. If the file does not exist, all is authorized.
The robots of the search engines will read this file regularly. If the large robots respect with the letter the directives, it should be known that others are unaware of prohibitions, or worse still use this file to be informed of the closed areas than they will excavate. It is not thus certainly a good manners to prohibit the access to certain robots. AuthorAnwar Hossain Author Emailanwtele@yahoo.com References
The Web Robots Pages
Search Indexing Robots and Robots.txt
robots txt Definitions to be consultedgoogle its the world famous search engine. Yahoo! Slurp Slurp is the name of the robot of indexing of the search engine Yahoo (initially Inktomi). YDS Yahoo! Desktop Search is defined as a retrieval software of information on the hard disks of the users. Thus, this "internal" search engine would make it possible to find the files mislaid on our computer. Yahoo Feed Seeker Yahoo Feed Seeker is the name of the Robot of indexing of flows RSS of Yahoo WOSE Search engine of objects, indexing not of the pages but of the objects, based on a technology of extraction of attributes.
The current version of test is specialized in the advertisements of the real estate. Thesaurus Search engine of objects, indexing not of the pages but of the objects, based on a technology of extraction of attributes.
The current version of test is specialized in the advertisements of the real estate. Taxonomy Science, laws, or principles of classification. Systematic. Division in ordered groups or of categories. Spamdexing Attempt at Spam specifically directed towards the search engines. Example: to try to position in the first positions of the results of an engine of the nonrelevant pages compared to the selected key word. MSNBot Robot of indexing of the search engines MSN.com and Live.com Boolean operator Possibility of binding several words of a request by one AND, one OR, one EXCEPT, etc. Tool of research Search engine directory or gate which posts in particular the results ("promotional bonds") provided by the Person receiving benefits in his pages, at the time of the request of a Net surfer. Examples: Google, Yahoo!, Earthlink, Club-Internet, Altavista, MSN France MSN.fr a general public Gate in France. It is published by Microsoft Tree structure Archie Software allowing to locate files on waiters ftp by requests on key words. Gmail Service of Webmail proposed by Google. AllTheWeb Alltheweb is an search engine Directory The directory, or Guides, or Repertory, is one of the systems of research most used on the Web.
|
