[ic] New bot user agent

[Jon Jensen]

On Thu, 27 Feb 2020, David Christensen wrote:

>>> This one I haven't seen before and was filling up my tmp directory:
>>>
>>> "Googlebot-Image/1.0"
>>
>> Interesting. Sounds like that would be a good addition to the robots 
>> list. Would you like to make a pull request against the GitHub repo?
>
> Wouldn’t the existing “GoogleBot” entry catch this, or are we (for some 
> reason) being case-sensitive/word boundary aware here?

Good point.

In the Interchange global structure file I see (trimmed for readability):

'RobotUA' => qr/gonzo|Google-Sitemaps|GoogleBot|grab/i,

so I would expect that to match.

DB, what does your interchange.structure file show for the RobotUA regex? 
You'll need to have DumpStructure Yes in your interchange.cfg for that 
file to be written at daemon startup.

Jon


-- 
Jon Jensen
End Point Corporation
https://www.endpoint.com/