Web


Summary Search Results Summary | Categories | Web Sites | Meta Search | Shopping

Web Site Results - Found 10 sites for robotstxt

1. The Web Robots Pages
The /robots.txt checker can check your site's /robots.txt file and meta tags. The IP Lookup can help find out more about what robots are visiting you.
http://www.robotstxt.org/ -
2. Robots exclusion standard - Wikipedia, the free encyclopedia
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from ...
http://en.wikipedia.org/wiki/Robots.txt -
3. The Web Robots Pages
This file must be accessible via HTTP on the local URL "/robots.txt". The contents of this file are specified below. This approach was chosen because it can be easily implemented ...
http://www.robotstxt.org/orig.html -
4. www.whitehouse.gov
User-agent: * Crawl-delay: 10 . Sitemap: http://www.whitehouse.gov/feed/media/video-audio
http://www.whitehouse.gov/robots.txt -
5. www.google.com
User-agent: * Disallow: /search. Disallow: /groups. Disallow: /images. Disallow: /catalogs. Disallow: /catalogues. Disallow: /news. Allow: /news/directory
http://www.google.com/robots.txt -
6. Robots.txt Generator - McAnerin International Inc.
robots.txt generator designed by an SEO for public use. Includes tutorial.
http://www.mcanerin.com/EN/search-engine/robots-tx ... -
7. Robots.txt Information
Information on the robots.txt and how it effects your website. Also includes a free robots.txt generator
http://www.robotstxt.ca/ -
8. Block or remove pages using a robots.txt file - Webmaster Tools Help
A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see ...
http://www.google.com/support/webmasters/bin/answe ... -
9. Introduction to "robots.txt"
Learn about the robots.txt, and how it can be used to control how search engines and crawlers do on your site.
http://www.javascriptkit.com/howto/robots.shtml -
10. Robots.txt and Search Indexing - Search Tools Report
Information on using the robots.txt file to keep web crawlers, spiders and robots from indexing certain sections of a site.
http://www.searchtools.com/robots/robots-txt.html -

Category Results - Found 0 categories for robotstxt
Sponsored


Refine Your Search



Home
(C) KOMCC Search Engine 2005
Contact Us