Since I had an interesting study object, I wanted to see how much I could uncover in my logfiles with a bit of cluster analysis. So I created a matrix from referrers and accessing IP addresses and got an overview of typical user scenarios - how do normal users look in the log, how do referrer spammers look, and how does our friend look.
All three variants can be distinguished well, even though I'd currently rather shy away from capturing it algorithmically - all of it can be simulated quite well. Still, a few peculiarities are noticeable. First, a completely normal user:
aa.bb.cc.dd: 7 accesses, 2005-02-05 03:01:45.00 - 2005-02-04 16:18:09.00
0065*-
0001*http://www.tagesschau.de/aktuell/meldungen/0,1185,OID4031994 ...
0001*http://www.tagesschau.de/aktuell/meldungen/0,1185,OID4031612 ...
0001*http://mudbomb.com/archives/2005/02/02/wysiwyg-plugin-for-wo ...
0001*http://www.heise.de/newsticker/meldung/55992
0001*http://log.netbib.de/archives/2005/02/04/nzz-online-archiv-n ...
0001*http://www.heise.de/newsticker/meldung/56000
0001*http://a.wholelottanothing.org/2005/02/no_one_can_have.html
You can nicely see how this user clicked away from my weblog and came back - the referrers are by no means all links to me, but incorrect referrers that browsers send when switching from one site to another. Referrers are actually supposed to be sent only when a link is really clicked - hardly any browser does that correctly. The visit was on a defined day and they got in directly by entering the domain name (the "-" referrers are at the top and the earliest referrer that appears is at the top).
Or here's an access from me:
aa.bb.cc.dd: 6 accesses, 2005-02-04 01:11:56.00 - 2005-02-03 08:27:09.00
0045*-
0001*http://www.aylwardfamily.com/content/tbping.asp
0001*http://temboz.rfc1437.de/view
0001*http://web.morons.org/article.jsp?sectionid=1&id=5947
0001*http://www.tagesschau.de/aktuell/meldungen/0,1185,OID4029220 ...
0001*http://sport.ard.de/sp/fussball/news200502/03/bvb_verpfaende ...
0001*http://www.cadenhead.org/workbench/entry/2005/02/03.html
I recognize myself by the referrer with temboz.rfc1437.de - that's my online aggregator. Looks similar - a lot of incorrectly sent referrers. Another user:
aa.bb.cc.dd: 19 accesses, 2005-02-12 14:45:35.00 - 2005-01-31 14:17:07.00
0015*http://www.muensterland.org/system/weblogUpdates.py
0002*-
0001*http://www.google.com/search?q=cocoa+openmcl&ie=UTF-8&oe=UTF ...
0001*http://blog.schockwellenreiter.de/8136
0001*http://www.google.com/search?q=%22Rainer+Joswig%22&ie=UTF-8& ...
0001*http://www.google.com/search?q=IDEKit&hl=de&lr=&c2coff=1&sta ...
This one came more often (across multiple days) via my update page on muensterland.org and also searched for Lisp topics. And they came from the shock wave guy once. Absolutely typical behavior.
Now in comparison, a typical referrer spammer:
aa.bb.cc.dd 6 accesses, 2005-02-12 17:27:27.00 - 2005-02-02 09:25:22.00
0002*http://tramadol.freakycheats.com/
0001*http://diet-pills.ronnieazza.com/
0001*http://phentermine.psxtreme.com/
0001*http://free-online-poker.yelucie.com/
0001*http://poker-games.psxtreme.com/
All referrers are direct domain referrers. No "-" referrers - so no accesses without a referrer. No other accesses - if I analyzed it more precisely by page type, it would be noticeable that no images, etc. are accessed. Easy to recognize - just looks sparse. Typical is also that each URL is listed only once or twice.
Now our new friend:
aa.bb.cc.dd: 100 accesses, 2005-02-13 15:06:16.00 - 2005-02-11 07:07:55.00
0039*-
0030*http://irish.typepad.com
0015*http://www208.pair.com
0015*http://blogs.salon.com
0015*http://hfilesreviewer.f2o.org
0015*http://betas.intercom.net
0005*http://vowe.net
0005*http://spleenville.com
What stands out are the referrers without a trailing slash - atypical for referrer spam. Also, just normal sites. Also noticeable is that pages are accessed without a referrer - hidden behind these are the RSS feeds. This one is also easily distinguishable from users. Especially since there's a certain rhythm to it - apparently always 15 accesses with one referrer, then switch the referrer. Either the referrer list is quite small, or I was lucky that it tried the same one with me twice - one of them is there 30 times.
Normal bots don't need much comparison - few of them send referrers and are therefore completely uninteresting. I had one that caught my attention:
aa.bb.cc.dd: 5 accesses, 2005-02-13 15:21:26.00 - 2005-01-31 01:01:07.00
2612*-
0003*http://www.everyfeed.com/admin/new_site_validation.php?site= ...
0002*http://www.everyfeed.com/admin/new_site_validation.php?site= ...
A new search engine for feeds that I didn't know yet. Apparently the admin had just entered my address somewhere beforehand and then the bot started collecting pages. After that, he activated my newly found feeds in the admin interface. Seems to be a small system - the bot runs from the same IP as the admin interface. Most other bots come from entire bot farms, web spidering is an expensive affair after all ...
In summary, it can be concluded that the current generation of referrer spammer bots and other bad bots are still quite primitive in structure. They don't use botnets to use many different addresses and hide that way, they use pure server URLs instead of page URLs and have other quite typical characteristics such as certain rhythms. They also almost always come multiple times.
Unfortunately, these are not good features to capture algorithmically - unless you run your referrers into a SQL database and check each referrer with appropriate queries against the typical criteria. This way you could definitely catch the usual suspects and block them right on the server. Because normal user accesses look quite different.
However, new generations are already in the works - as my little friend shows, the one with the missing slash. And thanks to the stupid browsers with their incorrectly generated referrers (which say much more about the browser's history than about actual link following), you can't simply counter-check the referenced pages, since many referrers are pure blind referrers.