Home arrow Search Engine Spiders arrow Page 3 - ROBOTS.TXT Primer
SEARCH ENGINE SPIDERS

ROBOTS.TXT Primer
By: Webby
  • Search For More Articles!
  • Disclaimer
  • Author Terms
  • Rating: 3 stars3 stars3 stars3 stars3 stars / 36
    2003-09-29

    Table of Contents:
  • ROBOTS.TXT Primer
  • Reasons To Use A ROBOTS.TXT File
  • ROBOTS.TXT Analysis

  • Rate this Article: Poor Best 
    SEARCH DEVARTICLES

    TOOLS YOU CAN USE

    advertisement

    ROBOTS.TXT Primer - ROBOTS.TXT Analysis


    (Page 3 of 3 )

    User-agent: EmailCollector
    Disallow: /

    If you were to copy and paste the above into notepad, save the file as robots.txt and then upload it to the root directory of your server (where you will find your home page)what you have done, is told a nasty email collector to keep out of your website. Which is good news as it may mean less spam!

    I do not have the space here for a fully fledged robots.txt tutorial, however there is a good one at
    http://www.robotstxt.org/wc/exclusion-admin.html

    Or simply use the robotsbeispiel.txt I have uploaded for you. Simply copy and paste it into notepad, save it as robots.txt and upload it to your server root directory.
    http://www.abakus-internet-marketing.de/robotsbeispiel.txt
    blog comments powered by Disqus

    SEARCH ENGINE SPIDERS ARTICLES

    - Search Engine Spiders: Why Don`t They Crawl ...
    - The Yahoo SLURP Crawler
    - How Search Engines Work (and Sometimes Don’t)
    - Spider Guts
    - Score One for the Spiders?
    - Protect Against Invaders by SPAM-Proofing Yo...
    - ROBOTS.TXT Primer
    - Designing Websites For Humans In A World Of...
     
    SEO Chat Forums  
     RSS  Articles
     RSS  Forums
     RSS  All Feeds
    Contact Us 
    Site Map 
    Request Media Kit
    Write For Us Get Paid 
    SEO Weekly Newsletter
     
    SEO Tools
    Adsense Calculator
    AdSense Preview
    Advanced Meta-Tags
    Alexa Rank Tool
    Check Server Headers
    Class C Checker
    Code to Text Ratio
    CPM Calculator
    Domain Age Check
    Domain Typos
    Future PageRank
    Google Dance
    Google Keywords
    Google Search
    Google Suggest
    Google vs Yahoo
    Indexed Pages
    Keyword Cloud
    Keyword Density
    Keyword Difficulty
    Keyword Optimizer
    Keyword Position
    Keyword Typos
    Link Popularity
    Link Price Calculator
    Meta Analyzer
    Meta Tag Generator
    Multiple Link Popularity
    Page Comparison
    Page Size
    PageRank Lookup
    PageRank Search
    Robots.txt Generator
    ROI Calculator 
    S.E. Comparison 
    S.E. Keyword Position 
    Site Link Analyzer 
    Spider Simulator 
    URL Redirect Check 
    URL Rewriting 
    Privacy Policy 
    Support 


    © 2003-2012 by Developer Shed. All rights reserved. DS Cluster 9 - Follow our Sitemap
    Popular SEO Chat Topics
    All Tutorials & Tools