[ic] Spiders and affiliate tracking

Jamie Neil jamie at versado.net
Wed Jan 7 16:11:31 EST 2004


Ed LaFrance wrote:

>> I can't stop the spiders following affiliate links, so I'm just trying 
>> to stop them from indexing the page they land on without affecting 
>> normal crawling behavoir.
> 
> 
> That will be tough. The spiders are probably following these links from 
> the affiliate sites and indexing as they go. You might try a [bounce] to 
> a page with no args when the landing page is called with an affiliate 
> code, that might prevent the index of the mv_pc string - but I am no 
> expert at these things, just a suggestions.

A "noindex" robots metatag on a page _should_ stop that page being 
indexed by a search engine when it is crawled (in fact it should remove 
the entry from it's index if already there). However I still don't like 
this as a solution.

After thinking about it some more last night, I've actually ended up 
using a bounce method like you mentioned. I've created a simple referral 
page:

pages/referral.html
-------------------
[tmp page][cgi name=ref][/tmp]
[if scratch page]
   [bounce page=|[scratch page]| status="301 Moved Permanently"]
[else]
   [bounce page=|ref_error| status="301 Moved Permanently"]
[/else]
[/if]

and an error page:

pages/ref_error.html
--------------------
[tag op=header]
Status: 404 Not found
Content-type: text/html
[/tag]
[tmp page_title]Sorry, the page was not found[/tmp]
@_NOLEFT_TOP_@
<H1>[scratch page_title]</H1>
<p>The requested page was not found. You can [page index]return to 
browsing our catalog</a>, if you wish.</p>
@_NOLEFT_BOTTOM_@

Instead of referrals using a URL like:

http://www.oursite.com/pagename.html?mv_pc=ref1

they will now use:

http://www.oursite.com/referral.html?mv_pc=ref1&ref=pagename

which sets the session source value for affiliate tracking and then 301s 
the user to the correct page:

http://www.oursite.com/pagename.html

AFAIK, Google (and other SEs) will not index a page that gives a 301 
error. Instead they will follow the redirect and index that page, thus 
ignoring the original referral URL. This should also stop people 
bookmarking affiliate links.

So far it seems to be working ok. Time will tell whether these new 
affliate links appear in the SERPs!

All I need to work out now is how to get the links that have already 
been indexed to get dropped.

-- 
Jamie Neil | <jamie at versado.net> | 0870 7777 454
Versado I.T. Services Ltd. | http://versado.net/ | 0845 450 1254



More information about the interchange-users mailing list