Content-type: matter-transport/sentient-life-form

AES (Rijndael) Encryption Test in JavaScript - AES Encryption in JavaScript

JavaScript Encryption Library - Blowfish Encryption in JavaScript

OpenPGP Message Encryption in JavaScript - AES and RSA and some support libraries in JavaScript - enables creating PGP-encrypted texts.

Rijndael in JavaScript - AES in JavaScript again, this time specifically designed for compatibility with older browsers as well.

twofish/javascript - TwoFish Encryption in JavaScript

code.enthought.com - Enthought Tool Suite - sounds like an interesting GUI library that builds on WxPython and enables even more comfortable application development. Particularly interesting is the use of the "Traits" concept for the automatic creation of interfaces.

/my/cssQuery/ - the IE7 author (not the one from Microsoft, but the compatibility hacker) has split out his CSS Query Engine as a standalone program. A very powerful way to find parts of documents using CSS selectors in JavaScript.

Aggregators and Referrer Spamming

Dave Winer I mean, with his newsRiver.root:

When the aggregator reads a feed it sets the referer for the request to http://www.newsriver.org/, a new site with a place-holder message for people who are unfamiliar with the aggregator. This address will appear in the referer logs for feeds that are read by users of this aggregator.

And again he spammed the referer, instead of adhering to the standard of putting the URL for the aggregator in the User-Agent header. No, not all hits from aggregators come from their pages, this is utter nonsense that he already perpetrated with the Radio Userland aggregator. And back then he showed himself to be forgetful when it was tried to explain it to him.

Why are programmers, especially in the field of web technologies and related topics, so incapable of looking at specifications? And simply adhering to what makes sense? The HTTP specification is not that complicated that one couldn't at least adhere to it rudimentarily ...

Codeville - and yet another version control system, this one is written in Python and specifically addresses the problem of merge algorithms.

monotone: distributed version control - by now Monotone is quite mature. Main advantage: it is distributed as a single executable. And the content-based addressing also sounds very interesting.

xmledit - A filetype plugin to help edit XML, HTML, and SGML documents : vim online - nice plugin for VIM (more precisely a macro package) to better edit XML files with tag completion and automatic closing and jumping between block start and end.

cucumber2: an object-relational mapping system for Python and PostgreSQL - another ORM for Python. Special feature here: PostgreSQL table inheritance is used to make the transitions between objects and classes easier. However, it is also not portable to other databases.

Lisp at Light Speed - a really interesting Lisp blog

Taste for the Web - a nice cartoon about Paul Graham's articles. Yes, his sometimes rather uncle-like style with constant plugs for Yahoo Stores can get on your nerves from time to time.

Re: Web application design: the REST of the story - a very interesting discussion of two currently dominant architectural styles for web applications: REST and Continuations.

LGT: Lightweight Game Toolkit for Python - particularly interesting are the NanoThreads (coroutines for Python), the EventNet (enhanced event programming) and Gherkin (an alternative to Pickle/Marshal). There is now an enhanced successor to NanoThreads and EventNet called FibraNet (which is independent of LGT).

Webstemmer - HTML-Grabber that extracts the actual core text from websites based on the layout.

simple_json 1.0 - Alternative to json.py with fewer quirks (therefore with a sociopath as a programmer - but you can't have everything. In this case, functional code is more important than friendly tone).

CSS2/DOM - Styling an input type="file" - wild hacks to style file upload buttons with CSS or JavaScript.

Arithmetic Games

I just realized that it makes more sense to calculate 90*24*60*60 instead of 90*24*60 - at least if you want to express 90 days in seconds. Now, comment cookies should also last longer than 1.5 days

StickBlog » Blog Archive » Upload multiple files with a single file element - a nice method to upload multiple files without having to deal with a forest of browse buttons.

Weblogs - Variation on the previous link, here JavaScript and CSS together.

Dejavu - Trac - another ORM for Python. This one is characterized by absurd class names (Arena, Sandbox, Units ...)

Download DrScheme v300 - a new version of the best Scheme system in the world is out. Grab it while it's fresh. Now with Unicode!

appscript - Python as an alternative to AppleScript. Thus, application control via the AppleScript interfaces directly from Python programs.

Generic Functions with Python

PEAK has been offering generic functions similar to CLOS for Python for quite some time. I always wanted to play around with it, but for a long time it was just part of PyProtocols, and the installation was a bit tricky. However, since September of this year, it has been decoupled and much easier to install. So I dove right in.

And I must say: wow. What Phillip J. Eby has accomplished is truly fantastic. The integration with Python (works from Python 2.3 - he even invented his own implementation of decorators for Python 2.3) is superb, even if, of course, some things take a bit of getting used to.

A small example:

import dispatch

[dispatch.generic()]
def anton(a,b):
 "handle two objects"

[anton.when('isinstance(a,int) and isinstance(b,int)')]
def anton(a,b):
 return a+b

[anton.when('isinstance(a,str) and isinstance(b,str)')]
def anton(a,b):
 return a+b

[anton.when('isinstance(a,str) and isinstance(b,int)')]
def anton(a,b):
 return a*b

[anton.when('isinstance(a,int) and isinstance(b,str)')]
def anton(a,b):
 return b*a

[anton.before('True')]
def anton(a,b):
 print type(a), type(b)

This small example simply provides a function called 'anton', which executes different code based on the parameter types. The example is of course completely nonsensical, but it shows some important properties of generic functions:

Generic functions are - unlike classic object/class methods - not bound to any classes or objects. Instead, they are selected based on their parameter types.
Parameter types must therefore be defined - this usually happens via a mini-language with which the selection conditions are formulated. This is also the only syntactic part that I don't like so much: the conditions are stored as strings. However, the integration is very good, and you get clean syntax errors already when loading.
A generic function can be overloaded with any conditions - not just the first parameter is decisive. Conditions can also make decisions based on values - any arbitrary Python expression can be used there.
With method combinations (methods are the concrete manifestations of a generic function here), you can modify a method before or after its call without touching the code itself. The example uses a before method that is always (hence the 'True') used to generate debugging output. Of course, you can also use conditions with before/after methods to attach to specific manifestations of the call of the generic function - making generic functions a full-fledged event system.

A pretty good article about RuleDispatch (the generic functions package) can be found at Developerworks.

The example, by the way, shows the Python 2.3 syntax for decorators. With Python 2.4, of course, the @ syntax can also be used. One disadvantage should not be kept secret: the definition of generic functions and their methods is not possible interactively - at least not with the Python 2.3 syntax. Unfortunately, you generally have to work with external definitions in files here.

RuleDispatch will definitely find a place in my toolbox - the syntax is simple enough, the possibilities, however, are gigantic. As an event system, it surpasses any other system in flexibility, and as a general way of structuring code, it comes very close to CLOS. It's a shame that Django will likely align with PyDispatch - in my opinion, RuleDispatch would fit much better (as many aspects in Django could be written as dispatch on multiple parameter types).

LTK - The Lisp Toolkit - if it should just be a bit of GUI, but not necessarily the big hammer is needed - LTK offers simple bindings for TK in Common Lisp. Works excellently with OpenMCL together, even CLISP likes it.

Sams Teach Yourself Shell Programming in 24 Hours - A whole book about shell programming. And of course, a pretty good introduction to the various tools that Unix systems provide. Certainly recommended for anyone who, for example, has gotten a root server and now wants to do more with it - but otherwise knows Linux mainly from the GUI.

[GOODIE] Headless Squeak for OS X (Re: Mac VM 3.2.X)](http://lists.squeakfoundation.org/pipermail/squeak-dev/2002-April/037668.html) - how to get a headless Squeak (Smalltalk environment without GUI component) running under OS X for server services. Particularly interesting for using Seaside.

Hyper Estraier: a full-text search system for communities - Full-text database with attribute search and some other nice features - as well as bindings for various programming languages

The Xapian Project - another full-text indexer, this one with various advanced features such as stemming for different languages.

Inets 2.5.5 - Webserver in Erlang

Is Rails a DSL? What is a DSL, and is it possible in Python? - Domain Specific Languages - a quite useful description and examination of the situation in Python and Ruby.

Linux Daemon Writing HOWTO - how to write a daemon under Linux (general information)

Yaws - another web server in Erlang - this one is HTTP 1.1 compatible and contains approaches for web development

Python Cheese Shop : python-fastcgi 1.0 - FastCGI implementation based on the OpenMarket FastCGI C library and therefore significantly faster than pure Python solutions.

Python OpenID 1.0.1 Released — OpenID Enabled - OpenID client and server in Python. I should check it out, could be quite interesting for comment functions.

Hacking the jProject - The Daily WTF - ouch. An order system where each order is stored in its own table in the SQL Server. Great idea.

How-To Guide for Descriptors - a very good explanation of how properties work in Python and what the magic methods get, set, and del are all about (and how getattribute plays into this).

Just a Thought

What would actually happen if the GNOME developers went to the Linux Kernel Mainling list and announced that they recommend users to use FreeBSD because the chroot model of Linux is pathetic, and the kernel APIs are a mess anyway, and Linux still doesn't have really good filesystem notifications, and the development of Linux simply doesn't take GUI requirements into account enough. Therefore, they would suggest users to use FreeBSD, because the Linux kernel programmers are all idiots anyway.

What would Linus' reaction look like?

And we all make the same mistakes again

There is currently a lot of activity in the area of Microformats - the idea behind it: to store information blocks not in XML, but in predefined HTML. CSS classes are then used to define what a single format is. Logically, there are problems with colliding styles - what a surprise. I myself am always amazed at how much energy developers can spend on stupid ideas.

We once had HTML that not only dealt with semantics but also with layout. And that produced the all-time favorite FONT-TAG orgies on HTML pages. Over time, most people have come to the realization that separating semantics and layout makes sense - semantics as a basis for marking up content, layout in the CSS files, and as a connection between these, the IDs and classes on tags. Additionally, with DIV and SPAN "anonymous" tags without predefined semantics (except "this is a block of text" and "this is an inline stretch of text" - where this meaning can be easily overloaded), for things that don't work with the normal semantic markup (which is mainly due to the rather stupid idea of HTML that there are markings for headings, but no markings for sections of text to which these headings would belong).

What do Microformats do now? Well, the same stupid idea of misusing something - namely in this case the connecting pieces between semantics and layout mentioned above. Microformats give these a meaning - for example, a DIV with a class 'description' would then be the description of a review - read the details in the hReview reference. Sorry, but this must inevitably lead to conflicts - have the idiots never heard of namespaces? The Microformats explicitly address XHTML - and that has exactly the purpose of embedding namespaces. And if you think you have to implement such a stupid idea - couldn't you at least be smart enough to give the parts more cryptic but unambiguous classes?

As I said, it's amazing how much energy goes into such stupid ideas that are doomed to create more problems than solutions.

Deadlock - interesting article about deadlocks in systems and about zombie processes, signal handling, etc.

setting user passwords in admin

A rather ugly - but still useful - monkeypatch:

# monkey-patch for auth.users
from django.models.auth import User

def user_pre_save(self):
 if not self.password.startswith('sha1$'):
 self.set_password(self.password)

User._pre_save = user_pre_save

Put this into your model file (or somewhere else that is loaded early on) and you will be able to set passwords in the admin by entering clear text passwords. If the password starts with 'sha1$', it is seen as already encrypted and nothing happens. If it doesn't start with that string, it is converted using the standard Django function for password encryption.

No, this isn't something that should go into core - it's far too ugly for that. But at least it allows you to set passwords through the admin, without requiring the user to calculate the actual password hash.

SystemExit and exception handlers

Frequently used: SystemExit. A Python exception that many people don't know. The special thing about this exception: it is not an error. It also does not occur unexpectedly. It is simply triggered by sys.exit. The idea behind this is that you can insert an end processing in the dynamic flow (e.g. some file cleanups), without linking into global exit processing (with all the problems that entails).

The problem is that many programs and libraries install a global exception handler. One that catches every error and sends it nicely formatted by mail, logs it somewhere or something similar. I do this all the time. It also works great - except when you actually want to initiate an early end in your program. Then nothing works anymore - because you get corresponding errors for a non-error.

This becomes particularly critical in connection with multiple processes. If you start a process during operation, you also want to terminate it without executing any subsequent code. You can best see this in an example program:

import signal
import os

try:
 pid = os.fork()
 if pid:
 print "Elternprozess", os.getpid()
 else:
 print "Kindprozess", os.getpid()
 sys.exit(0)
except:
 print 'Fehler aufgetreten in Prozess', os.getpid()

print "Das darf nur der Elternprozess ausführen", os.getpid()

This code simply has a global error handler that catches errors in a rather unspecific way. Within the code, a parallel process is started with fork. However, since SystemExit is treated like all other exceptions, the child process is not terminated correctly - a process copies the entire state of the parent process, including return addresses, open error handling, files, database connections and so on.

This is of course fatal - because here sys.exit is caught. So there is an error message for the quite normal sys.exit(0) call. And even worse: since SystemExit is not treated separately, it continues normally afterwards - and the child process runs into code for the parent process. Code runs double, which can have critical results under certain circumstances.

If you can fully control the entire software stack, the solution is simple:

import signal
import os

try:
 pid = os.fork()
 if pid:
 print "Elternprozess", os.getpid()
 else:
 print "Kindprozess", os.getpid()
 sys.exit(0)
except SystemExit:
 raise
except:
 print 'Fehler aufgetreten in Prozess', os.getpid()

print "Das darf nur der Elternprozess ausführen", os.getpid()

This simply re-raises the SystemExit - i.e. triggers it again - without making a message. In most cases, Python's standard handling will then kick in and convert the SystemExit into a normal termination.

But what to do if you have several stacked variants of the wrong error handling? I had something like this with Django and FLUP (the FCGI/SCGI server for Python). In Django I changed it, then the error hit in FLUP. What do you do then?

The solution is a bit more brutal:

import signal
import os

try:
 pid = os.fork()
 if pid:
 print "Elternprozess", os.getpid()
 else:
 print "Kindprozess", os.getpid()
 os.kill(os.getpid(), signal.SIGTERM)
except:
 print 'Fehler aufgetreten in Prozess', os.getpid()

print "Das darf nur der Elternprozess ausführen", os.getpid()

Ultimately, the process simply commits suicide - it sends itself a SIGTERM, i.e. a termination signal. The same one you would normally send from the shell. However, you must then ensure that any necessary post-cleanups are either already done, or then run in a SIGKILL handling routine - otherwise you may have problems (e.g. database transactions should already be committed).

With this solution, you also have to be careful that no open resources block the process - otherwise you may produce zombie processes. Often it is better for such multiprocessing to start a management process much earlier in the system - outside the error handling chain - and then use it to start processing processes. However, this then has the disadvantage that processes started in this way do not inherit the environment of the parent process. Therefore, you usually have to make more preparations to perform the desired actions. Incidentally, Apache pursues a similar approach - there the processes are created from a very early basic state, so that they come as resource-free as possible.

Vampire - An extension of mod_python that makes it more developer-friendly. For example, it can also perform automatic code reloading.

Learning Seaside - cool demo of what can be done with Seaside (Smalltalk web framework) and AJAX. Essentially a database interface with a freely configurable database model - something like Google Base, only cooler.

Ajax Sucks Most of the Time (Jakob Nielsen's Alertbox December 2005) - why Jacob Nielsen is right - sometimes.

Commentary - Sticky notes for websites, implemented as WSGI middleware. Very interesting, could be particularly interesting for source views or similar, or for longer texts.

pyinotify - very nice, finally a usable wrapper for the notify function in Linux. With it, Python programs can be informed about changes in the file system - ideal for directory monitoring.

Paj's Home: Cryptography: JavaScript MD5: sha1.js - JavaScript implementation of SHA1 - practical if you want to avoid plaintext passwords in web forms. Of course, you should always have a fallback, because not everyone has JavaScript available or activated. The site also has MD5 and MD4 implementations and a few other snippets on the topic.