[ic] RobotUA

Philip S. Hempel interchange-users@icdevgroup.org
Wed Nov 27 00:42:01 2002


Grant wrote:

>Thanks a lot for the info Phillip.  I'd like to clarify a couple things
>though...
>
>  
>
>>Usually with a 301 it takes a couple of runs from most spiders to decide
>>to go anywhere else into the
>>site.
>>    
>>
>
>What is the correct way to forward from your domain to your site's index
>page so the spiders don't get confused?
>  
>
Icdevgroup uses 302 and does not get indexed but for the first page, as 
you may have noticed, from google
most other search engines will tromp all over the site and produce gory 
listings
My opinion and of many others I have spoke with at 
http://www.webmastersworld.com (where the GoogleGuy hangs out) say 301 
is the only way if your going to redirect and expect rankings. You could 
use a doorway page
that uses java script and just place links into the sight with some 
keywords in it for the search engines, but feel that is
very unprofessional and spammy myself.


>  
>
>>Now depending on how long your system has been running with a 301
>>if you move now it will cause
>>you more problems. Realize that 301 is just like you told the mailman
>>you have a new address and then
>>you send a new change of address to all of your magazine companies.
>>Now how long does it take for them to get around to sending them to your
>>new address?
>>Then sundenlly you decide to send them and your mailman a new change of
>>address again even before
>>they have actually acted on your old change of address. Well you will
>>have at least 2 monthns before you get
>>any magazines or a good part of your mail will end up in different places.
>>
>>So usually using 301 in difference to 302 that says temp move don't keep
>>record of it. This is a very bad things
>>when it comes to spiders if you keep bouncing arround.
>>    
>>
>
>Are you saying a 301 or a 302 is better for spiders?
>  
>
Here is a link from Google that talks about what they feel you should do

http://www.google.com/remove.html

And the snippet that talks about 301

*Change the URL of your website*

Since Google's crawler associates the content of a page with its URL, 
there is no way to manually change the URL that is displayed for your 
website. The URL will be updated the next time we crawl your site. The 
crawler revisits each site according to an automatic schedule, and we 
cannot manually accelerate the date on which your site will be recrawled.

If the URL of your website has changed since we last crawled it, you may 
use the URL submission form <http://www.google.com/addurl.html> and the 
URL removal methods described below. However, the URL submission form 
does not take effect immediately, so using the URL removal feature may 
leave your website inaccessible from Google until we crawl your site again.

Instead of requesting a change from Google, we recommend that you ask 
the sites currently linked to your old site to update their links (to 
point to your new site). Also, don't forget to change any entries you 
may have in the Yahoo! directory and the Open Directory. Finally, if 
your old URLs redirect to your new site using HTTP 301 (permanent) 
redirects <http://www.ietf.org/rfc/rfc2616.txt>, our crawler will know 
to use the new URL. Changes made in this way will take 6-8 weeks to be 
reflected in Google.

I feel 301 is better.and also pay close attention to the time google 
says it will take for the crawler to understand the
new address (6-8 weeks)

>>This is spoken completly from experience since I did this myself and
>>have seen its effects.
>>
>>Also all of your DMOZ entries also need to point to your redirected
>>location to get credit for it.
>>
>>    
>>
>
>Where are these entries?
>  
>
If your site has been submitted to DMOZ at http://www.dmoz.org and since 
Google and other search engines use
these listing for supporting your rankings they should be set to always 
match your expected site location

>  
>
>>Point is this if you have just started doing this move, then leave it
>>alone. It will take at least 2 months for
>>google and a few others to catch up. If you have done this for awhile
>>you could completley lose at least
>>a months worth of crawls until they get around to seeing the new move.
>>
>>This happend to me and I got impatient myself and moved around again.
>>Lost much traffic and after talking to some people at webmasterworld,
>>they just told me to not mess with it and be patient they will crawl
>>your site within one to two
>>months. If your sids are not showing they will jump on it soon.
>>
>>--
>>Philip S. Hempel
>>debian/rules
>>    
>>
>
>It seems like there must be a better way to go about all this that doesn't
>use 301s at all so the spiders will head straight inside.  What would that
>be?
>
>- Grant
>
>  
>
Most SE's do not like 302 (temp redirect) and almost all suggest the 
usage of 301 (permanant redirect)
302 does not push ranking onto the main page since this is want you want.

Since I quit using 302 and went to 301 and a few other things I went from page 5 in
the rankings to number 1,2,3,4,5 for over 15 key word sets and went from 100 users
to over 800 users (not search engines) in a day average.

Goggle spiders over 200 pages on my site now and we have as of today on the average
have over 10 sales a day (from 1 every 3 weeks). (this is good for a supposed part time business)


Hope this helps if you need more ask.

(and please excuse typos, wrote this in a rush)
-- 
Philip S. Hempel
debian/rules