Ok, I've finished the plugin for Wordpress 1.5. Simple thing - a plugin and a small management page where you can set up various questions. To install you download the plugin and simply copy the files to the locations specified in the readme.txt and activate the plugin. Then you just add a few questions in the management section under Questionnaire and you're done. When commenting, a more or less silly question is asked, which should be satisfied with as short an answer as possible (we don't want to annoy the commenters too much). If the answer is correct, the comment - provided no other anti-spam methods kick in first - is released immediately. If the answer is wrong, the comment goes into moderation and must be approved by the admin.
You can of course also build a secret IQ test for your commenters with this and instead of simple questions put small riddles in there - only those who solve them are allowed to comment immediately.
I've activated the plugin on my site, let's see if it has any effects on the commenting behavior of people here. You can share your opinions here about what you think of such an anti-spam methodology.
A fairly interesting possible attack on any captcha solution can be found incidentally in the comments to Eric Meyer's WP-Gatekeeper: you can simply collect and save the comment forms. Additionally, you need a site where you can use these - for example, a site for free porn videos. There you present the captchas to the users of these sites and take their answers. You then send this answer to the saved form and the comment is done. Of course you can also take countermeasures against this - probably best would be an encoded timecode in the form and rejection of a timecode that's too old, since the answers from the porn viewers probably won't come immediately. Interesting approach, the whole thing.
Update: the plugin still has two bugs. For one, it also catches trackbacks (which of course never have the necessary variables) and it can currently still be circumvented pretty easily if you know what to look for in the form - you just need to solve one captcha and then you can spam other comments by changing the comment ID. The latter is actually a bug in many captcha solutions - you fall for it too easily, forgetting to bind the captchas to some form of serial number or similar so that a form can only be used once in that form...
So I'll be making an update to the plugin in the near future.
Update 2: the problem with trackbacks and pingbacks should now be solved. The problem with replay is still in there. I still need to think about that a bit. My previous solution approaches don't really appeal to me for that.
Update 3: I've now switched it off here again. I haven't gotten any comment spam so far and without a compelling reason, even a simple question to answer is pretty annoying...