[Info-vax] proper file format, attributes for non-binary files served by a web server

[Arne Vajhøj]

On 1/21/2012 11:20 AM, Phillip Helbig---undress to reply wrote:
> It would be nice if the access log of the web server provided an
> indication of whether something was a robot or not.  (For the common
> ones, one can see this from the name, of course.)  One could then use
> SEARCH to come up with a list of pages the search engines are hitting.

Impossible.

Anyone can write a crawler putting whatever they want in UA.

Arne