[ic] Yahoo/Slurp and caching of Session ID's - With Log data

Gary Norton gnorton at broadgap.com
Tue Mar 2 16:20:34 EST 2004


Ok, I have a bit more information and some new questions regarding the
Yahoo/Slurp caching problem referred to here:
(http://www.icdevgroup.org/pipermail/interchange-users/2004-February/037967.
html)

In my current log I get the following (see larger log snip below):
(compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

In my interchange.cfg I have both Yahoo and Slurp listed (separate entries).

Could the reason that session id's get cached be related to not having an
entry such as "Yahoo*Slurp", or something like that? As you can see from the
access snip below that each page is getting a different session ID
associated with it, and there are several different addresses requesting
pages.

Hopefully I can track this down soon.

-Gary

<<<<<<<<<<<<<<<<<<<<<<<<<< Larger Snip>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Here is my current interchange.cfg:
RobotUA <<EOR
    ATN_Worldwide, AltaVista, Arachnoidea, Aranha, Architext, Ask, Atomz,
    BackRub, Builder, CMC, Contact, Digital*Integrity, Directory, EZResult,
    Excite, Ferret, Fireball, Google, Gromit, Gulliver, Harvest, Hubater,
    H?m?h?kki, INGRID, IncyWincy, Jack, KIT*Fireball, Kototoi, LWP, Lycos,
    MegaSheep, Mercator, Nazilla, NetMechanic, NetResearchServer, NetScoop,
    ParaSite, Refiner, RoboDude, Rover, Rutgers, Scooter, Slurp, Spyder,
    T-H-U-N-D-E-R-S-T-O-N-E, Toutatis, Tv*Merc, Valkyrie, Voyager, WIRE,
    Walker, Wget, WhizBang, Wire, Wombat, Yahoo, Yandex, ZyBorg, appie,
    asterias, bot, contact, crawl, collector, fido, find, gazz, grabber,
    griffon, archiver, legs, marvin, mirago, moget, newscan, seek, speedy,
    spider, suke, tarantula, agent, topiclink, whowhere, winona, worm,
xtreme,
EOR

RobotIP <<EOR
    202.9.155.123,      204.152.191.41,         208.146.26.19,
    208.146.26.233,     209.185.141.209,        209.185.141.211,
    209.202.148.36,     209.202.148.41,         216.200.130.207,
    216.35.103.6?,      216.35.103.70,          66.196.65.??,
EOR

RobotHost <<EOR
    *.crawler*.com,     *.excite.com,           *.googlebot.com,
    *.infoseek.com,     *.inktomi.com,          *.inktomisearch.com,
    *.lycos.com,        *.pa-x.dec.com,         add-url.altavista.com,
    westinghouse-rsl-com-usa.NorthRoyalton.cw.net,
EOR


Here is a snip from the latest access log:
66.196.72.84 - - [02/Mar/2004:02:57:36 -0700] "GET
/cgi-bin/suscon/scsubs.html?category=U-bolts&id=6dBfjoQT HTTP/1.0" 200 32330
"-" "Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)"
66.196.90.20 - - [02/Mar/2004:03:03:08 -0700] "GET
/cgi-bin/suscon/scan/fi=products/st=db/co=yes/sf=category/se=Shims.html?id=X
qWdVJYM HTTP/1.0" 200 32972 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)"
66.196.90.181 - - [02/Mar/2004:03:09:07 -0700] "GET
/cgi-bin/suscon/scsubs.html?category=Lowering+Kits&id=oaxHtcWt HTTP/1.0" 200
31692 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)"
63.109.229.13 - - [02/Mar/2004:03:09:08 -0700] "GET
/cgi-bin/suscon/sway_bars.html HTTP/1.0" 200 39765
"http://search.yahoo.com/search?p=f150+sway+bars&sp=1&ei=UTF-8&n=20&fl=0&fr=
fp-tab-web-t" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; COE June
03, 2002; .NET CLR 1.0.3705)"
63.109.229.13 - - [02/Mar/2004:03:12:36 -0700] "GET
/cgi-bin/suscon/scan/fi=products/st=db/co=1/sf=category/se=Sway%20Bars/op=eq
/nu=0/sf=veh_make/se=Ford/op=eq/nu=0/ml=25/tf=category/to=x/tf=veh_make/to=x
/tf=description/to=x.html?id=VDdK2pdI HTTP/1.0" 200 54070
"http://search.yahoo.com/search?p=f150+sway+bars&sp=1&ei=UTF-8&n=20&fl=0&fr=
fp-tab-web-t" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; COE June
03, 2002; .NET CLR 1.0.3705)"
66.196.90.43 - - [02/Mar/2004:03:12:47 -0700] "GET
/cgi-bin/suscon/scan/fi=products/st=db/co=1/sf=category/se=Coil%20Spring%20S
pacers/op=eq/nu=0/sf=veh_make/se=Jeep/op=eq/nu=0/ml=25/tf=category/to=x/tf=v
eh_make/to=x/tf=description/to=x.html?id=WkwVdwja HTTP/1.0" 200 54304 "-"
"Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)"
66.196.90.43 - - [02/Mar/2004:03:17:17 -0700] "GET
/cgi-bin/suscon/scan/fi=products/st=db/co=1/sf=category/se=Helper%20Springs/
op=eq/nu=0/sf=veh_make/se=Toyota/op=eq/nu=0/ml=25/tf=category/to=x/tf=veh_ma
ke/to=x/tf=description/to=x.html?id=MWxpEaKN HTTP/1.0" 200 32348 "-"
"Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)"
66.196.90.126 - - [02/Mar/2004:03:17:55 -0700] "GET
/cgi-bin/suscon/scsubs.html?category=Lowering+Kits&id=GUmKUNcc HTTP/1.0" 200
31692 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)"
66.196.90.130 - - [02/Mar/2004:03:18:16 -0700] "GET
/cgi-bin/suscon/scsubs.html?category=Axle+Pivot+Brackets&id=JVbwKetB
HTTP/1.0" 200 29877 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)"
66.196.90.68 - - [02/Mar/2004:03:18:35 -0700] "GET
/cgi-bin/suscon?id=Mk98ejAN HTTP/1.0" 200 39521 "-" "Mozilla/5.0
(compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
66.196.90.73 - - [02/Mar/2004:03:18:51 -0700] "GET
/cgi-bin/suscon/scsubs.html?category=Helper+Springs&id=3R3HX4GS HTTP/1.0"
200 32818 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)" 

<<<<<<<<<<<<<<<<<<<<<<<<<< /Larger Snip>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
--------------------------------------------------------------------
Gary Norton
broadGap Technologies
http://www.broadgap.com





More information about the interchange-users mailing list