[interchange-cvs] interchange - racke modified debian/robots.cfg

Mon Apr 7 08:02:04 2003

User:      racke
Date:      2003-04-07 12:00:22 GMT
Added:     debian   robots.cfg
Log:
robots configuration file added

Revision  Changes    Path
1.1                  interchange/debian/robots.cfg


rev 1.1, prev_rev 1.0
Index: robots.cfg
===================================================================
RobotUA <<EOR
    ATN_Worldwide, AltaVista, Arachnoidea, Aranha, Architext, Ask, Atomz,
    BackRub, Builder, CMC, Contact, Digital*Integrity, Directory, EZResult,
    Excite, Ferret, Fireball, Google, Gromit, Gulliver, Harvest, Hubater,
    H?m?h?kki, INGRID, IncyWincy, Jack, KIT*Fireball, Kototoi, LWP, Lycos,
    MegaSheep, Mercator, Nazilla, NetMechanic, NetResearchServer, NetScoop,
    ParaSite, Refiner, RoboDude, Rover, Rutgers, Scooter, Slurp, Spyder,
    T-H-U-N-D-E-R-S-T-O-N-E, Toutatis, Tv*Merc, Valkyrie, Voyager, WIRE,
    Walker, Wget, WhizBang, Wire, Wombat, Yahoo, Yandex, ZyBorg, appie,
    asterias, bot, contact, crawl, collector, fido, find, gazz, grabber,
    griffon, archiver, legs, marvin, mirago, moget, newscan, seek, speedy,
    spider, suke, tarantula, agent, topiclink, whowhere, winona, worm, xtreme,
EOR

RobotIP <<EOR
    202.9.155.123,      204.152.191.41,         208.146.26.19,
    208.146.26.233,     209.185.141.209,        209.185.141.211,
    209.202.148.36,     209.202.148.41,         216.200.130.207,
    216.35.103.6?,      216.35.103.70,
EOR

RobotHost <<EOR
    *.crawler*.com,     *.excite.com,           *.googlebot.com,
    *.infoseek.com,     *.inktomi.com,          *.inktomisearch.com,
    *.lycos.com,        *.pa-x.dec.com,         add-url.altavista.com,
    westinghouse-rsl-com-usa.NorthRoyalton.cw.net,
EOR