Planet CompSoc

So I've finally tracked down why Royal Mail's website looks horrible in Firefox on Linux. They sniff user agents and if it isn't one of their 'supported' browsers then they send different HTML that doesn't render properly.

Not only are they incapable of writing web pages that work across multiple browsers, they actually check for Firefox on Windows. If you have Firefox on Linux, which has the exact same Gecko engine, then you get sent the HTML which doesn't render properly. If you spoof your user agent, then all is fine.

Not only is this a big oversight, but it seems silly to go to all the trouble to sniff the user agents, if the HTML they send to unsupported browsers doesn't render correctly in any web browser anyway?

 I have sent an email, but don't expect a reply, nor that they'll heed my suggestions.

When working on Telepathy, I've often wanted to be see which D-Bus methods are being called on whom, when signals are emitted, and so on. Timing information is also handy: I'd like to figure out why cold-starting Empathy takes 12 seconds, and it'd be much easier if I could look at a diagram rather than staring at the unreadable output of dbus-monitor.

Previously, Alban wrote a tool that used a patched version of mscgen, and produced appropriate input with a dbus-monitor-like Python script. I wanted some more D-Bus-specific diagrams, and ended up reimplementing both the monitoring component (by forking dbus-monitor, as its --profile output did not contain quite enough information) and the diagram-drawing component (using Cairo). I'm happy to present an initial release of Bustle:

There's a Telepathy-specific hack in the tool to shorten object paths, but it shouldn't make the tool any less useful for looking at other D-Bus traffic.

I haven't made binary packages yet, I'm afraid, so you'll need to grab the source tarball and build it if you want to try it out. In Debian-land, the dependencies are libdbus-1-dev libglib2.0-dev libghc6-mtl-dev libghc6-cairo-dev libghc6-gtk-dev libghc6-parsec-dev; see README in the source tree for how to build and use it.

The astute among you may have noticed from the dependencies that the diagram-building component is implemented in Haskell, using the excellent bindings to Gtk+ and Cairo. I got a prototype going within a few hours, and the strong correctness guarantees that the type system provides meant that I could refactor it mercilessly with confidence. I'm sure that I would have spent many frustrating hours chasing type bugs had I written it in Python, which is a more conventional high-level language for prototyping and writing tools like this. Next time you're frustrated by such bugs, you should give Haskell a try. :-)

I’ve not been sleeping all that well lately. And let’s be honest, this probably has something to do with being in the middle of a term at Oxford University, with a lot on my mind and far too much to see to.

However.

I’ve noticed recently that my room contains a surprising number of light sources, which I suspect can’t be helping me get to sleep. Reading from left to right, we have:

• Light leaking under the door from the staircase (where the lights are apparently permanently on)
• Blue power LED flashing on my laptop
• Green power/charging LED on my other laptop
• Huge red light on the multiplug under my desk
• Flashing lights on my hub

I’ve therefore come up with a cunning plan:

• Block the base of the door with an old pillow
• Prop a book over the laptop light
• Put the other laptop in a drawer
• Tie an old sock round the multiplug
• Shove the hub into a cardboard box with holes for the wires

Whether or not this helps remains to be seen. But here’s a plea to manufacturers: please think twice about the brightness and location of your LEDs, especially if there’s a chance someone will be trying to sleep in the same room as the device they’re on.

I finally got round to putting Linux on my Advent 4211 yesterday. There are various ways and means of doing this; the ones which worked for me were:

• Use the installation image optimised for small screens
• Follow these instructions to make a bootable USB drive from the image (note: it’s ‘cat’, not ‘zcat’ as the ubuntu image is not zipped. I know they’re Debian instructions, but they seem to work)
• Use the script given in this forum post (run as root) to get the WiFi working.

The end results are really rather impressive; here’s a pretty picture:

Here’s one for all you progammers out there: what happens to the values of x and y if you type the following in, say, Python?


>>> x = 5
>>> y = 6
>>> x,y = y,x


I’d always half-thought, without really considering it, that you’d end up with x=6 and y=6, i.e. that the above was syntactic sugar for:


>>> x = y
>>> y = x


Actually, though, it turns out…


>>> x=5
>>> y=6
>>> x,y=y,x
>>> print x,y
6 5


In what onlookers are already calling “potentially unwise”, our robot overlord ordered various toys from ThinkGeek.

Today — being as it is Friday, always the most productive of all days — they arrived! Apparently appropriate tools to control the missile launchers are not yet packaged for Debian. :(

VMware do seem to be moving towards a whole "woo look at my management console" thing.
Turns out (despite my misgivings) that it's quite possible to do USB passthrough of mass storage devices to your guest OS - so, for example, running XP in a VM, you plug in your Stick O' USB, and have XP use it rather than your Debian Etch host machine.  This assumes you're running Server 2.x, and so have Webby Goodness.
Turns out that this is a really simple trick.
Step 1: Ensure that you're in the plugdev group (46).
Step 2: Modify the permissions with which /proc/bus/usb is mounted:Yep, that's right kids, on Etch /proc/bus/usb is no longer mounted by /etc/fstab.  So don't bother putting your entries there, it won't work.
Step 3: Reboot
Step 4: Profit (yeah, I know this should be step 3).

If it doesn't work - you do have a USB controller installed on your VM, don't you?

If you're anything like me, you've probably missed appointments due to your bad memory. Well now there's now excuse, you can use your trusty basic phone to recieve reminders with reminder calls from Asterisk.

My new script, AsteriCal, will poll your Google Calendar's iCal URL and call you to remind you of upcoming events. Find out more here

It's written in Python, and communicates with Asterisk by placisng call files in Asterisk's outgoing calls spool directory.

You may have heard of the site Say No To 0870. It's a UK database of mappings of Company Names and their 0870/0844/0845 numbers to their geographic equivalents. Using the geographic equivlalents is a good idea - they're cheaper and included in a lot of plans' free minutes. Some companies can detect people calling their geographical numbers. Prefixing the number with 141 to withhold your number sometimes beats the system; sometimes it doesn't.

I thought it'd be a good idea to integrate the Say No To 0870 database with my Asterisk PBX, so that when I dial a number beginning with 08, Asterisk will actually call the geographical number with my Trunk provider. I achieved this using Asterisk AGI and Python. I screenscraped the Say No To 0870 website (theres no API, nor did they reply to my email) and untangled the HTML with Beautiful Soup.

If an 08 number has no entry, it just gets dialled. If there is an entry, it presents the user with a looping menu asking them to press 1 for the geographical number or 2 to dial the 08 number (in case the destination is smart, or the Say No To 0870 database is wrong). The script only considers the Main (Verified) numbers in the site's blue search results box.

You can view the script or, in Debian, you can install like so:
(Other distros store agi-bin in different places):

cd /usr/share/asterisk/
mkdir agi-bin
cd agi-bin
wget http://www.crummy.com/software/BeautifulSoup/download/BeautifulSoup.py
wget http://www.maniacmartin.com/files/noto0870.agi
chmod +x noto0870.agi
cd ../sounds
wget http://www.maniacmartin.com/files/mm-foundacheapernumber.gsm

Now, you need to add the following to the contexts in your /etc/asterisk/extensions.conf where you wish to allow users to dial 08-prefixed numbers:

exten => _08[4578].,1,AGI(noto0870.agi|${EXTEN})          ; Lookup in saynoto0870
exten => _08[4578].,2,Macro(dial-uk,0044${EXTEN:1})       ; If no match or user rejects, use normal UK trunk

The line for other UK numbers shouldn't accept 08 now. Something like this will do:

exten => _0[1237]XXXXXXXXX,1,Macro(dial-uk,0044${EXTEN:1})  ; Normal UK numbers

If a user chooses to use a cheaper number, the script will change the extension and jump to it on exiting.

How to setup Festival with Asterisk for arbitrary text-to-speech in your PBX.

Whilst playing with asterisk, I wanted the feature of text-to-speech, so the PBX can read things such as caller IDs.

The information at voip-info.org would have been helpful, except they couldn't reproduce the conf file verbatim as their in-house wiki parses ((double-bracketed text)) as links, so to help you, I've got a verbatim config file here:

(define (tts_textasterisk string mode) 
"(tts_textasterisk STRING MODE) 
Apply tts to STRING. This function is specifically designed for 
use in server mode so a single function call may synthesize the string. 
This function name may be added to the server safe functions." 
(let ((wholeutt (utt.synth (eval (list 'Utterance 'Text string))))) 
(utt.wave.resample wholeutt 8000) 
(utt.wave.rescale wholeutt 5) 
(utt.send.wave.client wholeutt)))

In Debian, the Festival conf file lives in /etc/festival.scm. Paste this into it, then restart Festival using its init script. Now in Asterisk's extensions.conf, you can call it like so:

exten =>2,1,Festival(Hi\, you are being called by)
exten =>2,n,Festival($CALLERID)

Don't forget to escape commas and such. (OK, this was a bad example, as you'd just use SayDigits() in this case)

My normal approach is useless here.

Do you write Python (not very well?).

Ever wondered why your calls to os.execvp() and its ilk don’t work as expected?

The answer may be that you’re doing this:

os.execvp('ls', ['/home/fred'])

When you actually mean this:

os.execvp('ls', ['ls', '/home/fred'])

As ever, reading the Python docs (help(os.execvp)) is usually better than randomly googling for an answer.

I was recently reading Regulation of Investigatory Powers Bill -- Some Scenarios and scenario 2 about session keys perked my interest. I wondered if it could be done with GnuPG, and after researching for a while I discovered it can, and here's how: As you probably know, in semi-new UK law, RIPA means that the police can demand that you hand over the plaintext of an encrypted file, or in certain circumstances demand that you hand over a key of your choice that decrypts the file. Since public/private key encryption is several orders of magnitude slower than symmetric encryption, GnuPG generates a random symmetric key and encrypts the plaintext with this. The symmetric key is then encrypted with the public key of the recipient and tacked on the front of the output. So, if the police want a key to a file, using your private key you can retrieve the symmetric key used for this particular file, and by handing this one-time key over you have satisfied the requirements under RIPA without compromising your main private key and allowing law enforcement to decrypt further messages without your knowledge or sign messages as you. To get the one-time key, do this on an uncompromised machine:

$ gpg --show-session-key accomplises.asc

You need a passphrase to unlock the secret key for
user: "Martin Smith "
1024-bit ELG-E key, ID 941AAA41, created 2006-10-23 (main key ID 50CB07BC)

gpg: encrypted with 1024-bit ELG-E key, ID 941AAA41, created 2006-10-23
      "Martin Smith "
gpg: session key: `9:12D785497247CA3C929EC62EAB63BD8A8F0BB29E9B60CF11FC43788B197F6A11'

Now law enforcement can use this key to decrypt accomplises.asc without needing any public or private keys:

$gpg --override-session-key 9:12D785497247CA3C929EC62EAB63BD8A8F0BB29E9B60CF11FC43788B197F6A11 accomplises.asc
gpg: encrypted with 1024-bit ELG-E key, ID 941AAA41, created 2006-10-23
      "Martin Smith "

The cycle ride is over, and I'm in the process of collecting those pledged sponsorships. People have an amazing ability to not have any money on them when its time to pay up, which is pretty convenient for them. Anyway, I figured that I will not be updating the website any more, since all of the days are already blogged. Due to the heavy GPX caching and the fact it was my first Django (and web framework for that matter) usage, which I chopped and changed many times, keeping this dead site online was costing me 71MB of RAM, which is quite a sizable proportion of my 256MB VPS. I decided that the best way was to make a static snapshot of the site, and serve it with my regular httpd directly. It was basically a matter of getting the snapshot: wget -nH -m http://lejog.maniacmartin.com And renaming the GPX files and their references by running these commands from the appropriate directories: for i in $(ls|grep -v index); do sed -i s@/posts/gpx/$i/@/posts/gpx/$i.gpx@g $i/index.html; done for i in $(ls|grep -v index); do mv $i/index.html $i.gpx; rm -r $i/; done I then copied the static/ and media/ folders over. I also added a mime type for GPX files for cleanliness to /etc/mime.types: application/gpx+xml gpx (You could also use the more generic text/xml type) I know for a Django site, this leaves a mess of folders all containing a single index.html page, but since the site will never be updated again, it frees up RAM and CPU cycles (caching of parsed GPX is intensive), and means that the site will load faster. Given the heavy usage of Google Maps, any speed up is a bonus.

One of the topics that popped up repeatedly at DjangoCon last weekend was how bad purely normalised table structures are, and how denormalisation is good for many things, including making your database cry less.

To that end, during the gaps in PyConUK this weekend, I decided to see how easy it would be to write a new Django field that will automatically denormalise a field in a related table across to another model.

Let's take an example; imagine we have this kind of model:

class Picture(models.model):
    user = models.ForeignKey(User)
    user_username = DenormField(User, "username")
    image = models.ImageField(upload_to="foo")
    title = models.CharField(max_length=100)

The idea is that the user's username (which, let's face it, you're going to need almost every time you need the photo) is stored in the same table, saving you from doing that join to get it from users each time. While this may not seem like much, it can be useful, and besides, it was fun to write.

My DenormField simply listens to a trio of the relevant signals, and then updates this value whenever it detects that the related User object has been updated, so it only costs on write (which, let's face it, is much better than on every read). You can do this kind of copying in normal Django app code as well, but I think this solution is much cleaner. I would do, I guess, since I wrote it.

It's only really a little test exercise in writing Strange New Fields (and I wanted to see how much monkeypatching it would take - none, as it turns out, just some very nasty code tricks). You can get a copy of it if you feel like having a play (warning: only briefly tested; may attach captions to your cat).

There are not very many things that could tempt me to fly 10 hours twice for only a weekend, but DjangoCon was one of them, and I'm glad I went; it was one of the best weekends I've had for quite a while.

There were plenty of good talks; I can't mention them all here, since as we all know, bytes cost money, but in that vein I must at least mention Cal Henderson's talk 'Why I Hate Django' - a brilliantly done talk on the things Django's missing, as well as the fact we don't have a mascot (or do we?) and that we aren't smug enough.

Still, I enjoyed many of the other talks as well (what a shame it is not to be able to see them all in person; still, Google will hopefully release the videos soon), as well as our Schema Evolutions panel, which seems to point towards a bright future for Django and migrations. 

Talking of bright futures, judging by the final session with Adrian and Jacob, there's plenty of stuff to get working on for 1.1, some of which have me quite excited. Multiple database support is going to be one of those important ones, really, but then Malcolm seems to have snuck the bottom layer of it in already, which is quite handy.

All in all, an excellent weekend, and finally a chance to meet a subset of everyone involved in Django (and they really do come from many places), and I can now hardly wait to get over to Prague next year for the promised EuroDjangoCon. If you didn't make it, then I highly recommend watching the videos when Google release them (after editing them on their ridicously great AV system, no doubt).

The world of migrations in Django is definitely warming up now, and with that we're proud to announce the release of South 0.3, the 'intelligent django migrations app', available now from south.aeracode.org.

New in this release are the use of fields rather than dicts for specifying columns in migrations, so migrations now look cleaner, work with all the databases Django supports to a better extent, and even support custom fields.

Also new is dependencies between apps, for those situations where your forum app's Post depends on your accounts app's Profile; South will work out the right way to apply everything so foreign keys don't horribly break.

There's also a nice number of bugfixes and general tweaks Andy and I have been making as South moves towards being used in active development work. If you think you might be needing migrations, or even if you don't, give South a try!

(Also new is a very carefully worded "why we're best" page on our wiki. I'm trying not to step on people's toes, just say why we're using South rather than anything else, but I'm not sure how well that's working)

With the launch of its own site - south.aeracode.org - and Andy McCurdy jumping in to help out, South has been making some good progress. We've made a few backwards incompatable changes, which while annoying will hopefully make things much easier in future.

Firstly, the format of the create_table and add_column calls has been changed to use Fields instead of long dicts of parameters. As well as looking more Django-like, this allows us to reuse the SQL generating features Django already has for fields, as well as playing nicer with custom types.

In line with these changes, startmigration has also seen some changes; as well as generating things using Fields now, it can group models together in migrations (this proves useful for foreign keys referencing tables made after them).

We're still working on finishing this all off, and there'll hopefully be something stable to show off soon. Woop.

Also, if you're going to DjangoCon, may I recommend the Schema Evolutions panel; there'll be all manner of evolutiony and migrationary things discussed therein.

I bought myself an Advent 4211 last week (you might know it as the MSI Wind; they’re essentially the same thing). So far, it’s lived up to expectations; there are just a couple of things to say…

• Mine seems to have the decent Synaptics touchpad - read The Register’s review for the warning about the other one.
• You really need to use headphones with it if apps you’re listening to don’t have a very good sound level - inevitably in a device this size, the speakers aren’t great, especially at full volume.

How the Linux install on it goes is something I shall let y’all know shortly…

My laptop contains an ATI Radeon chip.  Yes, I know that this is Bad for Linux-compatability - I was young and inexperienced when I got it.  Said laptop also runs Debian testing/unstable (/experimental), and used the xserver-xorg-video-radeon driver.

Now, kernel versions > 2.6.22 have been causing it to freeze completely when starting X - no network response, no nothing (with a lack of serial port, the trick is to boot into rescue mode, start ssh, ssh in, then continue to boot while tailing logs).  This is Less Than Ideal, especially given the number of security holes that have been discovered that apply to my 2.6.22 install.

So, last night, I decided to see what the fglrx driver was up to these days.  Last time I tried it, it had a nice tendency to hang the machine if I did, erm, pretty much anything, especially switching away from VT7.  Imagine my surprise when, having installed the appropriate packages (apt-cache search fglrx IYF) and prodded xorg.conf, it Just Worked™ - well, apart from the fact that mouse and keyboard input didn't seem to be happening...  Turns out that the xserver-xorg-input-kbd, xserver-xorg-input-mouse (and in my case xserver-xorg-input-synaptics) are kinda useful.  So, a quick switch to VT1, install, restart gdm, and switch back (look Ma, no hanging!), and I have working X.  Hooray!

If only the random router/firewall issues at work would sort themselves out as simply...

I may also be missing the most fabulous Jude in the world, and so have to keep busy somehow...

For the next two weeks or so I'm helping Martin Smith do his Land's End to John O'Groats trip, and so I'll only have slow internets to communicate from. Still, you can read about our travels at the new, shiny, lejog blog.

While this means you won't be able to send me post (the old, snail-paced variety) for a while, I doubt any of you want so, so that's not really important.

Also, once we get back there'll be a load of GPS data I will set to work visualising, if you like that kind of thing, and burning questions (which I have yet to determine) will be answered.

After the initial release some time last week, and a generally positive reaction, South now has a 0.2 release.

The most important (and only major) change is the addition of MySQL support, so the other section of Django users can finally give it a try. There's also the ability to create all models' migrations at once.

The main South project page has also been slightly improved, and while I've added even more text it should hopefully be easier to grok quicky.

If you have any feedback or advice on South, remember to tell me! If I don't know about something, it might not happen...

Regular readers will know that I recently migrated to lighttpd from Apache 2, to try and enable my blog to handle traffic spikes better. Out of curiosity, I decided to benchmark the new setup. I used a dedicated server on a 100Mbit line at ovh, France to perform the tests on my 256mb Xen VPS in London. First I installed ApacheBench, which is in the Debian package apache2-utils. Then I made it send 1000 requests, 10 concurrently, (figures I chose off the top of my head) to each of the following sites I host:

• A static file
• Wordpress with wp-cache plugin on php5-cgi with 4 processes under fastcgi
• Django eshop running in threaded mode under fastcgi
• CherryPy webpwman running under fastcgi

Now I don't claim it was a fair test, but the results are pretty shocking.: Average time per request on ion.maniacmartin.com:

Ping time	12ms
Static	53ms
Django	113ms
CherryPy	238ms
Wordpress	1700ms (+539 failed requests)

When my blog was reddited, it was Wordpress without wp-cache on Apache workers, all of which contained unncessesary mod_python Django instances. It stood no chance. Even after the optimisations I have done, Wordpress still performs shockingly bad compared to the alternatives. I will probably be switching my blog to a Django-powered backend, such as the soon-to-be-released aeblog, that powers Aeracode. Update: You can get a second opinion, which also shows that Django is faster, but by a much smaller margin.

After learning that my 256mb Debian Xen VPS died instantly after being reddited, I decided to take some action to prevent the same thing happening again. My blog is currently powered by Wordpress, so I enabled the wp-cache plugin that I had installed some time ago. It is however, one of the few PHP-powered things on my server. Then I decided to migrate to lighttpd, because it uses less memory than Apache, and because I like changing things on my server for fun. There were some hurdles, but I got there in the end. Here's what I'd do now if I had to do the migration again. They are not necessarily the best settings for all sites. I chose these settings as each site is small, and I don't have much CPU or RAM to play with. I use vim as my text editor. If you prefer something else (nano, emacs etc), then obviously substitute where necessary:

• Install lighttpd - apt-get install lighttpd . Then edit /etc/lighttpd/lighttpd.conf and set the port to 81. This will let us test lighttpd whilst Apache is running. Run apt-get install lighttpd again and it will finish successfully this time.
• Enable lighttpd modules - I need fastcgi for PHP, Django and CherryPy. Enable it like so: cd /etc/lighttpd/conf-enabled ln -s ../conf-available/10-fastcgi.conf 10-fastcgi.conf aptitude install php5-cgi The Debian release has support for PHP4 by default. Change this to PHP5: sed s/php4/php5/g 10-fast-cgi.conf I also wanted ssl support: ln -s ../conf-available/10-ssl.conf ssl.conf vim ssl.conf Specify your pem file location. If you have only a key and crt, cat them together into a .pem. Note that you cannot test lighttpd with Apache running if you have the SSL module enabled, as they'll compete for port 443
• Specify vhosts - You probably host multiple websites, and want the equivalent of Apache virtualhosts: I have one file for web framework-powered sites, and one for sites that serve PHP, but its up to you: vim vhosts.conf This specifies a vhost's URL with a regex. In this case its middlerasenchurch.org.uk or any subdomain of it.

  $HTTP["host"] =~ "(^|\.)middlerasenchurch\.org\.uk" {
          server.document-root = "/var/www/middlerasenchurch.org.uk"
  }

• Rewrite .htaccess files - lighttpd doesn't understand Apache's .htaccess files, but you can specify rewrites if the rewrite module is enabled (uncommented) in /etc/lighttpd/lighttpd.conf. If you need it, go and uncomment it. These rules make Wordpress work on my site:

  $HTTP["host"] =~ "(^|www\.)maniacmartin\.com" {
          server.document-root = "/var/www/maniacmartin.com",
          alias.url = (
                  "/svn" => "/usr/share/websvn/"
          )
          url.rewrite = (
          "^/?$" => "/blog/index.php",
          "^/blog/(wp-.+)$" => "$0",
          "^/blog/xmlrpc.php" => "$0",
          "^/blog/sitemap.xml" => "$0",
          "^/blog/(.+)/?$" => "/blog/index.php/$1"
          )
  }
  

• Make lighttpd spawn CherryPy - I used fastcgi to link lighttpd and CherryPy. First, I added this code to my python project, making it serve on fastcgi if it got passed the "--fastcgi" switch:

  if "--fastcgi" in sys.argv:
  	app = cherrypy.tree.mount(WebPWMan())
  	# CherryPy autoreload must be disabled for the flup server to work
  	cherrypy.config.update({'engine.autoreload_on':False})
  	cherrypy.config.update({
  			"tools.sessions.on": True,
  			"tools.sessions.timeout": 5,
  			"log.screen": False,
  			"log.access_file": "/tmp/cherry_access.log",
  			"log.error_file": "/tmp/cherry_error.log",
  	})
  	from flup.server.fcgi import WSGIServer
  	cherrypy.config.update({'engine.autoreload_on':False})
  	WSGIServer(app).run()
  	
  else:
  	cherrypy.config.update({'server.socket_port':8051})
  	cherrypy.config.update({"tools.sessions.on": True, "tools.sessions.timeout": 5})
  	cherrypy.quickstart(WebPWMan())
  

  Its important to redirect the logs away from stdout, as stdout is whipped away from CherryPy when you quit SSH, and the process will die. Also, logs should go to a folder that www-data can write to. (/tmp is probably not a good idea, but I don't read these logs anyway) In a file in /etc/lighttpd/conf-enabled, add an entry like this:

  $HTTP["host"] =~ "webpwman\.maniacmartin\.com" {
    fastcgi.server = (
      "/" => (
        "script.fcgi" => (
          "bin-path" => "/home/martin/webpwman/webpwman.py --fastcgi",
          "socket"          => "/tmp/webpwman.sock",
          "check-local"     => "disable",
          "disable-time"    => 1,
          "min-procs"       => 1,
          "max-procs"       => 1, # adjust as needed
        ),
      ),
    )
  }
  

• Configure Django in lighttpd - I run Django as seperate processes, and only pass it dynamic URLs, as lighttpd is much faster at serving static files. You could also do this with CherryPy, but my CherryPy site (webpwman) only has one static file, so it wasn't worth it. My confs look something like:

  $HTTP["host"] =~ "(^|\.)littledoves\.co\.uk" {
          server.document-root = "/var/www/littledoves.co.uk"
          fastcgi.server = (
                  "/littledoves.fcgi" => (
                          "main" => (
                                  # Use host / port instead of socket for TCP fastcgi
                                  # "host" => "127.0.0.1",
                                  # "port" => 3033,
                                  "socket" => "/tmp/littledoves.sock",
                                  "check-local" => "disable",
                          )
                  ),
          )
          alias.url = (
                  "/media/" => "/usr/local/django/django_src/django/contrib/admin/media/",
                  # Change to match where you put Django
          )
  
          url.rewrite-once = (
                  "^(/media.*)$" => "$1",
                  "^(/static.*)$" => "$1",
                  "^/favicon\.ico$" => "/static/favicon.ico",
                  "^(/.*)$" => "/littledoves.fcgi$1",
          )
  }
  

• Make an init script for your Django projects I started out with the script at http://code.djangoproject.com/wiki/InitdScriptForLinux, saved in /etc/init.d/django-fastcgi I changed the central part to:

  start-stop-daemon --start --chuid $RUN_AS --pidfile $RUNFILES_PATH/$SITE.pid --startas /usr/bin/python $SITES_PATH/$SITE/manage.py runfcgi method=threaded socket=$RUNFILES_PATH/$SITE.sock pidfile=$RUNFILES_PATH/$SITE.pid
                         pidfile=$RUNFILES_PATH/$SITE.pid
              chmod 400 $RUNFILES_PATH/$SITE.pid
  

  and changed the parameters at the top to: #### SERVER SPECIFIC CONFIGURATION DJANGO_SITES="lejogger littledoves" SITES_PATH=/usr/local/django/ RUNFILES_PATH=/tmp RUN_AS=www-data #### DO NOT CHANGE ANYTHING AFTER THIS LINE! This makes it use sockets, not TCP/IP ports. I used /tmp, not /var/run, as it needs to be a folder that www-data (or whatever user you run your Django instances as), can write to. To make the init script run on startup: chmod +x /etc/init.d/django-fastcgi update-rc.d django-fastcgi
• Test everything - I can't stress this enough. You know what to do Dont forget to restart lighttpd though: /etc/init.d/lighttpd restart While you're at it, you might want to specify different log files per virtual host too.
• Go live - First we need to disable Apache from starting on boot: cd /etc/rc2.d mv S91apache2 K09apache2 Change lighttpd to port 80 and go: /etc/init.d/lighttpd stop vim /etc/lighttpd/lighttpd.conf /etc/init.d/apache2 stop; /etc/init.d/lighttpd reload

Its all going fine here, apart from that I've lost mod_svn, which lighttpd doesn't have. I've switched to svn+ssh, and plan to add some features to the PHP-based websvn and deploy it soon.

With the recent massive changes to Django trunk - newforms-admin being the biggest - I've found a need to run both oldforms-admin and newforms-admin versions in both development and production; here's a quick HOWTO on both.

Development is arguably easier, although at times more annoying. The easy way, and the one I used, is the one blogged about by Patrick Altman, where you have a small script to change what your Python sees as the django module, which you can run each time you change project to a different version. His uses subversion checkouts every time; I prefer the one below, which chooses one of a set of already-checked-out Djangos, allowing quicker (and offline) changing:

#!/bin/sh
sudo rm /usr/lib/python2.5/site-packages/django
sudo ln -s /home/andrew/Programs/django-$1/django /usr/lib/python2.5/site-packages/django

Obviously this needs changing for wherever you keep your code; however, I have two folders in ~/Programs, django-trunk and django-0.97, so I just run something like "djangoswitch 0.97".

Production is a little trickier, but cleaner. Since you need to run more than one site at once, you can't go around relinking the django directory. However, all you have to do instead is tell mod_python (if you're using that method of deployment) you want a different Django version with a line like this:

PythonPath "['/usr/local/django/django_1.0'] + sys.path"

where all your other PythonBlah lines are in your Apache config. (Note, on my server the global version is 0.97-pre-svn, so I have a special override to 1.0 for my one app that uses it currently - this blog).

FastCGI is somewhat harder, but LastGraph doesn't need upgrading yet. I imagine it might take something like virtualenv to do it (or edit the manage.py script to twiddle with sys.path at the start).

webpwman is an online password manager that I wrote in CherryPy, that can import from KDE pwmanager CSV exports, and run behind an SSL-enabled webserver (which also prevents MITM attacks). It asks 3 security questions, which it randomly rotates on every correct login and asks for a master password which is used to decrypt the password from a json file. The idea being, that if you're on a compromised public terminal, then the bad guys should only get the passwords you viewed that session. It needs no database engine and supports only a single user, as I don't want to endorse storing your passwords with third parties, although I need such a tool as I can't remember my randomly generated passwords when out and about. I'd like to take this moment to congratulate the CherryPy team - I really like it, but your deployment documentation wasn't thorough enough for me, and I had to do a lot of googling to get it to sit behind another web server. Overall I've tried to keep it simple, and I'd appreciate any feedback, criticism or feature requests, especially regarding vunerabilities, as this is my first CherryPy project. Download and install webpwman here »

Tired of having to drop tables to re-syncdb your django models? Django-evolution not working, or too magic? Then I have just the solution; my newest project, South; intelligent migrations for Django projects.

Invented out of a frustration with continuous syncdb taunting, South can deal with migrations that suddenly vanish or appear, is designed to have database-agnostic migrations, and can autogenerate migrations based on your models.

It's certainly not yet complete, with only one database in the so-called database abstraction layer, and not all the fields working yet, but it powers this site, so I have proof it works! Take a look at the project page, or have a glance through the tutorial.

And please, send me some feedback! Or even better, patches, but that's highly optimistic :)

While I have been a happy user of Wordpress for many years, it was about time I jumped ship and moved my blog to something that let me manage things like project lists more sanely, and so welcome to my new Django-powered blog.

It uses a blog application/framework/thing I built from the ground up; while that's kind of unremarkable, given how easy this kind of stuff is if you've got the power of Django, one nice feature is that the URLs are a superset of the old Wordpress URLs. I even have a script that tests all the URLs from my server logs to help prove this. Honestly. (I may have missed some in the output, though, after it started testing attempted hacks)

Also of note is the tumblelog-ish thing on the side of the homepage, which just combines my feeds from last.fm, twitter and delicious (for now; more might appear soon). Hopefully it will make my near-glacial blogging speed a little less obvious.

I might well be releasing the code at some point soon, although I have to release the migrations library it depends on first! In the meantime, enjoy marvelling, examining or criticising my three evenings' work.

I recently started writing an online password manager. The basic idea is that it would ask 3 questions from a bank of questions, and then prompt for a decryption password and the name of a service of which to get a password. The service would run over SSL, and the passwords would be exported from my laptops KDE pwmanager in my server backup script. That way I'd be able to get my unmemorable passwords from any internet-connected terminal, without fear of an eavesdropper or piece of malware getting all of my passwords, since the 3 questions would change, much like the digits asked for when using online banking. Django seemed a touch too heavy, and I was tempted to use PHP, but Andrew suggested I give web.py, the slim web framework behind reddit, a try. web.py is indeed very slim, and I soon realised I'd have to find a different database ORM, as web.py's involved writing raw SQL as arguments, which I didn't really want to do. So I downloaded SQLObject. Midway through the project I decided that I needed sessions, so I didn't have to authenticate on every page. The web.py sessions example code calls web.ctx.session, but that didn't seem to exist for me. The web.py cookbook's page on sessions calls a web.session.Session(..) but specifies that this is available on version 0.3 only, which is strange, since the Get It! link on the site goes to a file called web.py-0.23.tar.gz, which seems to imply a latest version of 0.23. Eventually, I tried to find this session code myself, but it didn't seem to exist.

martin@zapper:webpwman/web $ grep -R session * martin@zapper:webpwman/web $

Anyway, back to SQLObject. I already had a postgresql database hanging around from when I was developing lejogger, So I thought I'd use that for developing webpwman (yes, I'm that lazy). The database is set up so that the user 'lejogger' can access it without any password. (ie psql -Ulejogger lejogger works fine), so I defined its URI: "postgres://lejogger@localhost/lejogger" This only produced tracebacks complaining about the lack of a password, and eventually I realised it was probably trying to connect via TCP/IP rather than the socket file, so I changed the URI to: "postgres://lejogger@/var/run/postgresql/lejogger" But it seems the leading / on the socket's file path disappears (This happens no matter how many I insert into the URI). In fact, I can see no way to make SQLObject connect to a postgresql database file socket whilst overriding the default username of the UNIX user, and I don't really want to edit my database permissions. After consideration, I have decided to rewrite the application using CherryPy (whose sessions code hopefully exists), and use a json file to store the data, since there won't be that much of it anyway and the passwords will be encrypted. This will also make it easier to import data, as the backup script can make then rsync/scp the json file up to the server. If I have missed something glaringly obvious, please contact me and I will update the post accordingly.

I guess it's just fussy to want not-too-hot and not-too-wet - at least it's not flooding.

Also, while the following works for cable modems:



You are strongly advised not to do so to your custom firewall (appropriate for Enterprise, apparently), unless you feel like digging out an old machine, a handy Debian install disk, network cards and your iptables knowledge and missing lunch...
On the other hand it might solve an issue where new VMware ESX machines (and, in fact, new physical machines) are added to the network and can't talk outside the local subnet.  Had we not cannibalized our Virtual Centre machine to get a network card for the firewall, I'd test it...

(also, I miss compressedchaos :()

Still, it's Saturday, and I have internet again \o/  (Oh, and hello Planet OxIRC, which I may actually be on now - that has probably about tripled the readership of this...).

This post is by no means up to the standard of the official Django 1.0 alpha release notes; it merely details what I had to do to get a Django 0.96 site working under Django 1.0 alpha. First, update your Django code. Mine's under svn, but svn up spewed an error, so i deleted the folder and checked out Django again. Next, open the models.py file in each of your applications. Remove the Admin classes from each model. Also, remove the prepopulate_from parameters from fields. Now open urls.py, and remove the old entry: # Uncomment this for admin: # (r'^admin/', include('django.contrib.admin.urls')), and replace it with (r'^admin/doc/', include('django.contrib.admindocs.urls')), (r'^admin/(.*)', admin.site.root), (the first line enables admin documentation). Also, add to the top of the urls.py file: from django.contrib import admin admin.autodiscover() admin.autodiscover() will look for an admin.py module in each installed app in INSTALLED_APPS and incorperate it into an admin page. Now, for each app that we want admin functionality for, make its admin.py The bare minimum you can do is simply inherit from the class they provide. You can do it like so: Extract from appname/admin.py:

from django.contrib import admin from breeze.web.models import * class CountryAdmin(admin.ModelAdmin): pass admin.site.register(Country, CountryAdmin) class UserLevelAdmin(admin.ModelAdmin): pass admin.site.register(UserLevel, UserLevelAdmin) class UserProfileAdmin(admin.ModelAdmin): prepopulated_fields = {"slug": ("name",)} # Populates slug field from name field admin.site.register(UserProfile, UserProfileAdmin)

Now, when you type ./manage.py runserver it should just work. Of course you can do lots more magic , see the Django Admin Documentation for more things, such as inline displaying of related tables. You can even make multiple admin classes.

I have recently embarked on writing a simple open-source BitTorrent private tracker and community website, as a single Django project. I felt compelled to do this after seeing the maze of PHP code behind other BitTorrent tracker/community sites, such as TBDev and xbtit. I want breeze to be simple yet functional, and easy to understand. I'm currently porting what I've written to Django 1.0 alpha, as initial development was done in the more familiar 0.96 release. As usual, you can check out my code using subversion (David-style joke) with svn co https://www.maniacmartin.com/svn/breeze You might also be interested in the (amazingly short) at tracker specification

I'm going to the Cambridge Folk Festival this coming long weekend (Thursday 31st July–Sunday 3rd August). The person I was camping with is probably not going to make it, having just had an operation; does anyone want her ticket? The face value is a hundred British pounds; she'll take substantially less. Let me know by the medium of your choice if you're interested!

LastGraph has been fixed, at least as far as I can tell; the last.fm team fixed the API bug pretty quickly on Monday, so thanks to them. If your profile was affected (that is, if its timeline is flat), email me and I'll delete its data file. Finally, I would have ended with a cheap phrase like "Keep on LastGraphing!", but that would have been cheesy and silly.

It seems that the launch of last.fm's new site has strangely broken the older API that LastGraph uses. I knew I was going to have to update to the new API, but I didn't think it would be this soon! For the two eagle-eyed people who've contacted me about this, well done - I'm working on a fix now (the precise problem is that the old API now always returns the last week's data, no matter when you ask for, hence the very flat graphs). Update: After a quick three minute URL change, it seems the new one is ALSO broken, and other things such as the lovely Last.fm Spiral are broken too. Time to go make sure they know about this at last.fm HQ.

I've decided to cycle the length of the UK for fun (and charity as well). And what better way to get in the mood by trying to learn Django and make a semi-useful Python app. So, I've given it a shot with a GPS/Google Maps mashup thing to analyse blogged route parts from maemo mapper on one of Andrew's Nokia internet tablets. Obviously you won't be able to see all of that until after we have actually blogged something - so check back after Monday, 12th August to see live progress (if it all works that is). It also accepts pledges of sponsorship, you can donate, whilst browsing the source code in subversion. Before anyone tells me, I know I'm not the best programmer in the world and that I took corners. Its made available only in the hope it'll be interesting or useful. You can check out more details at Martin's Land's End to John O'Groats cycle ride.

It comes to my attention that Jonathan Coulton is putting his clogs back on and coming back to the UK in the autumn. His first show here was great fun; you should go out of your way to see him this time around! Tickets are on sale for a show at Shepherd's Bush Empire on October 30th, which is the city I'm most interested in going to. I'll buy some tickets in the next day or so, so if you want me to get you one (and thus sit with a mostly-known set of people) let me know!

(Oxford people might prefer to see him there two days later, but I'm moving out in 10 days. Sniff.)

I’ve been aware for some time that my DNS isn’t quite as securely configured as I’d like. http://crashrecovery.org/named/ looks pretty good, but the two main issues bugging me were:

1. Anyone could do a ‘dig @ns.dnorth.net dnorth.net AXFR’ to retrieve a listing of all my DNS records - not great from a security point of view. This is a capability that should only be turned on for secondary DNS servers which need to fetch from the master.
2. The server would perform arbitrary lookups [for any domain] on request. This means it’s operating in ‘recursive mode’, which is a Bad Thing for various reasons.

The solutions were:

1. Add “allow-transfer { “slaves”; };” (without the double quotes) to the section of the configuration beginning “zone ‘dnorth.net’”. Then add a section defining the “slaves” access control list to be the local server, plus the secondaries: “acl slaves { 127.0.0.1; 123.45.67.89; }” replacing 123.45.67.89 by the IP address(es) of your secondary nameserver(s).
2. Add “recursion: no;” to the “options” section of the configuration.

Then restart the BIND9 service - on Debian, this is “/etc/init.d/bind9 restart”.

Health warning: Don’t do (2) above if you rely on your server to do its own DNS resolution - follow the crashrecovery tutorial above instead.

Well, last night I gave a talk at one of the ever-brilliant Oxford Geek Nights, and in case you're baying for the slides I used (all seven of them), you can find them at this wonderful OGN7 LastGraph Slides link. If you live within punting distance of Oxford, you should really try to come to the next one on August 27th. If not; well, they put videos up on the site... In a related note, I am enjoying the improvement on my slide-making capabilities via Inkscape. Previously, PDFs exported to massive file sizes and tying the individual pages together wasn't easy. Now, the new Cairo export in Inkscape (the same PDF export library LastGraph uses, yay) makes reasonably-sized PDFs, and pdftk munges them together easily and in record time. One day I'll give in and use an actual presentation program.

I find the different onomatopoeias used for the same sound in different languages entertaining, not least for the ensuing hilarity when you mispronounce them as English words. When I was in Paris with two of my housemates in December, we found a postcard which serves as a perfect illustration of why I enjoy mispronunciation. According to the back of the card, it depicts “Le coming-out du cochon”.

In the same shop, we found another card featuring a painting of a startled-looking boy playing his accordion, with a monkey dancing on his shoulder. I can’t decide how best to interpret the monkey’s expression.

If you’re not a massive nerd, I was just shown a joke which you might enjoy more than the rest of this post:

What do you call the blood that comes out when teenagers cut themselves?

Emoglobin.

---

On Thursday, Compsoc had a lightning talk session. Four people spoke:

• Andrew, the President, spoke about LastGraph (slides);
• I talked about ikiwiki, which we use behind the scenes for Compsoc’s website;
• Luke gave a talk about virtual worlds in general and Second Life in particular, and gave a demo;
• David spoke on “Windows Vs. Linux: Which Is Better?”, featuring a hilarious sketched-and-scanned graph which I would like a copy of.

I’m more impressed with Second Life than I thought I’d be. If only my laptop were speedy enough to run the thing, I’d take a look. (I tried to run it on my i855; the CPU pegged itself then everything died. Hard reset time!) I like the idea of a scripting language where everything is a state machine. It’s a bit of a shame that the only way to get code into or out of Second Life is to copy-paste between the built-in Notepad-alike editor and your proper editor of choice, which presumably does new-fangled things like “version control”. Oh well.

In case you care what I said about ikiwiki, I stuck my slides on the ’tubes. I’ve figured a few things out since the talk, such as how to get post-commit hooks working in Mercurial. Dom has some plans to add branching to Ikiwiki, so that you can make a test instance of the wiki to play with a new plugin just by making a new branch in svn or whatever. I like this idea.

S5 is pretty slick. I’d not used it before. (I didn’t actually use it directly, though: I relied on Pandoc to convert a file written with [Markdown] into an S5 slide show.) Anyway, there’s a lot to be said for having a slide show that you can just fire up in any web browser. (Except Andrew’s Konqueror apparently didn’t understand it. I wonder whether this is S5’s fault or Pandoc’s fault.)

Pandoc is a highly slick piece of software. It converts from any of Markdown, reStructuredText, HTML or L^AT_Eχ to Markdown, rst, HTML, L^AT_Eχ, man pages, ConTeXt, DocBook, RTF, or S5. It does all kinds of nice things along the way like curlifying quotes, making proper en-dashes, rendering L^AT_Eχ expressions as images so you can use it inline in Markdown and have it exported properly as HTML, and so on. It turns out to be a better Markdown processor than markdown itself, and supports extra things like tables, footnotes and so on. It turns out that the output from a practical I finished today was very nearly valid Pandockian table markdown, so I got a well-formatted project report pretty much for free. Oh, and for extra winning, it’s written in Haskell. I like the idea of being able to write man pages and documentation in something other than raw groff or DocBook XML.

I think the lightning talk format worked pretty well for Compsoc; it strikes me as a good way to get people involved. Nice work, committee!

The camel has two humps^[pdf] was an interesting read; a bit wooly, but believable. One of my favourite paragraphs, tangential to the paper's findings, concerned IDEs:

Programmers, who on the whole like to point and click, often expect that if you make programming point-and-click, then novices will find it easier. The entire field can be summarised as saying “no, they don’t”.

Being a very visual person, I naturally wanted a nice way of visualising the traffic lastgraph gets (especially as I nurse it through the initial round of bugs, most of which have gone). The ever-wonderful gltail always helps, but the same author has gone one further and come up with gltrail: It draws paths between pages visited, and looks pretty nice when visualising the lastgraph logs. It's also incredibly tempting to put it on a large screen in my room, the only problem being that I lack a large screen. And enough room. In a related note, it would be nice if there were some sort of AJAXy, real-time, web-based statistics like gltail; anyone know of any? Someone must have put that <canvas> element to good use by now...

Yes, this is my new official term for the server going up and down. It seems that some part of the system causes 100% CPU usage on the box it's running on, and in fact is so nasty to the system that it even makes the serial console lag. This, for those who are not server geeks, is Bad. I'm on the case, and have brought the full force of monitoring tools to bear to see if I can track down the cause. In the meantime, please accept my apologies for any downtime.

Yes, this is my new official term for the server going up and down. It seems that some part of the system causes 100% CPU usage on the box it's running on, and in fact is so nasty to the system that it even makes the serial console lag. This, for those who are not server geeks, is Bad. I'm on the case, and have brought the full force of monitoring tools to bear to see if I can track down the cause. In the meantime, please accept my apologies for any downtime.

After much work, lastgraph3's beta is now out. Please be warned that it may not work, may explode, and may abduct your cat, dog or goldfish in the process of failing. Apart from that, please feel free to give it a try at lastgraph3.aeracode.org. Report any and all problems back to me, either here, or at my email or something similar. Oh, and it's also very slow at fetching until I re-negotiate fast fetching with last.fm, since I've moved IP. Update: I've decided to make this one of those public, neverending betas, so LastGraph is now basically open for anyone and everyone to use. The last feature I added was LastGraph Premium, my new way of soliciting donations (this time, you can donate, and in return get a few extra features). I'm still not sure about it, but I've had some emails before from people who would have liked it, so we'll see how it turns out.

Something that's been bugging me for a while is that when I updated my Wordpress install (using svn), file uploads ceased to work. Attempting to upload a file resulted in the message "An error occurred in the upload. Please try again later.". It seems that the latest Wordpress uses Flash to show a progress bar during the upload. This site details many possible solutions, but the easiest one that's guaranteed to work is downloading the No-Flash-Uploader plugin. Simply drop it in your wp-content/plugins folder, then activate it in the Dashboard, and your uploads are fixed.

Something that’s been bugging me for a while is that when I updated my Wordpress install (using svn), file uploads ceased to work. Attempting to upload a file resulted in the message “An error occurred in the upload. Please try again later.”.

It seems that the latest Wordpress uses Flash to show a progress bar during the upload. This site details many possible solutions, but the easiest one that’s guaranteed to work is downloading the No-Flash-Uploader plugin. Simply drop it in your wp-content/plugins folder, then activate it in the Dashboard, and your uploads are fixed.

Having just put Windows XP back (in a dual boot configuration) I wanted to share the password manager that I have in the KDE system tray with Windows. I looked on the internet, and the only thing that has been ported is the pwmanager_dump program. However, combined with some the Windows ext2 driver, GNUWin32 tools (grep, awk, xargs etc) and a program that copies its command-line argument to the Windows clipboard, I have a hacked-up working pwmanager that syncs with linux.

I’m not 100% certain, but I think all of the tools I use are under the GPL, except for CopytoClipboard.exe, whose license I do not know. To get it to work, simply edit the first line of the bat file to point to your pwd file - of course you’ll have used the Windows ext2 driver to mount your linux partition to a Windows drive letter.

PwManager in Windows

It really isn't, honest. I'm working on the final part - the graph scheduler, which replicates the previous queued graph functionality - and I'll hopefully be pushing it all live soon. I was tempted to just push the live artist browsing part live, which all works currently, but it's only interesting for, oh, the first few hours...

I did a lightning talk to CompSoc last night on whether 64bit is ready for use on the desktop - if you’re bored, you can download the slides and the second of my famous sketch and scanned graphs.

gq has been a necessary evil in my life for some time now. I need a graphical LDAP client for use on the CompSoc systems, but gq (to be fair, the versions of gq packaged for Ubuntu) seems to be very buggy, segfaulting all over the place if you try to do anything other than browse with it.

Last week, after upgrading to a 64bit version of Ubuntu for the first time, I finally ditched gq, after running into identical symptoms to this Debian bug.

The good news is that there’s an alternative that actually works: it’s called Luma.

Have fun.

or, How to disable Bug Buddy in recent Gnomes

Apparently Bug Buddy is now a Gtk module. This is great because now every Gtk program gets it for free, not just libgnome applications. But it turns out to be kind of annoying when you're hunting a GHC bug using a test case that uses Gtk2Hs, and it keeps popping up in your face, crying that it doesn't know how to report bugs in the test case.

Some rummaging revealed that it's enabled by gnome-session setting GTK_MODULES=gnomebreakpad at login, so you can export GTK_MODULES= to shut it up.

Update: Sjoerd Simons pointed out that setting GNOME_DISABLE_CRASH_DIALOG=1 is probably what I actually mean. There might be other things in GTK_MODULES which you actually want, so blowing them away might be detrimental to your health.

---

(Brought to you by the why does Google not give me a useful answer to the search disable bug buddy? department.)

Whilst I’m not one to cover a news story that’s already been covered elsewhere, TorrentFreak has a priceless quote from Virgin Media’s CEO: “This net neutrality thing is a load of bollocks”.

Virgin seem to have no problems in admitting that they are throttling sites that do not pay Virgin extra cash, despite the fact that these sites have already paid their own hosting providers.Read the full story here, and if you’re a customer, ring up, moan and consider switching

People who know me, at all, will know I love two things: cheese, and the colour blue. Oddly enough, I don't like blue cheese, but that must be some kind of cancellation effect. Until the availability of web-based cheese, I have instead opted for sticking more blue in the site. Unless you're using one of those newfangled 'arr-ess-ess readers', you'll have noticed by now. In other news, I'm working on an upgrade to lastgraph3. I've finally had a sudden onset of common sense and have modified the downloader to not be 'scalable' and so not fail, am storing data in pickles rather than a database (it's way faster and more efficient, since the data is denormalised). There's also a more interactive interface, including browsing of artist-specific listening histories and weekly artist breakdowns. Hopefully it'll be out by the end of the week, but who knows, what with some exams apparently in the way.

If you don't know or care about IPv6, you probably want to

Have you ever noticed that the Boddingtons logo features a pair of bees? I'd vaguely spotted them before, but until earlier today, when someone pointed it out while buying one, it hadn't occurred to me that they're pretty incongruous. What association with bees does Boddingtons have? The only other beer with bees in its logo I know of uses honey as an ingredient (and is much tastier than Boddingtons).

The boring explanation is probably that the Boddingtons company colours have always been black and yellow, or something. Why those colours? We just don't know. But I'd like to think that it's actually a bad pun on the Poddington Peas.

It's kind of a long story, but I have a massive pile of stamps of various denominations. Returning a keyboard to Amazon cost £6.50, so... I used a few of them.

Warning: this post may contain rants about computers.

LDAP is, in principle, a very shiny thing. Single database of all your users, contacts, etc, meaning you don't have to hack stuff to keep passwd files, etc in sync.

However, there are a few pitfalls.

#1: make sure your database is indexed

I added some more 'index eq' entries to my slapd.conf, including 'index uid eq' and 'index cn eq'. This should help improve search speed (I think), but has a minor drawback - after restarting slapd, it seemed not to like me.

 m@z ~ % ldapsearch -x uid=michael uid
 m@z ~ % 

Yep, that's right, it returned absolutely nothing.
Turns out that you need to stop slapd and run slapdindex (tip: it'll probably screw up all the permissions so go chown openldap:openldap /var/lib/ldap/*). Restarting it then gives joy!

 m@z ~ % ldapsearch -x -LLL uid=michael uid
 dn: uid=michael,ou=users,dc=michaelh,dc=hopto,dc=org
 uid: michael

 m@z ~ %

#2: SSL/TLS issues

The Debian slapd package has switched from OpenSSL to GNUTLS (although in what can really only be described as epic failure, some of the stable packages are linked against one, while some are linked against the other - bug 457182). That's something of an aside, though - I don't know if it's causing the current issues or not (I know that it has been causing CompSoc some problems, but someone else will be taking that over soon).

Anyway, the current issue requires a little background. There are two machines - call them a and z. Now, a is running slapd, which listens on localhost:389 (ldap) and localnetworkip:636 (ldaps). Both machines are set to use LDAP for user lookups, etc. They have the same versions of libnss-ldapd, libpam-ldap, ldapscripts, libldap2, libldap-2.4-2. /etc/ldap/ldap.conf on z has:

URI ldaps://a/

and on a:

URI ldaps://a/ ldap://localhost/

Now, here's the fun bit. ldaps connections from z work:

 m@z ~ % ldapsearch -x -H ldaps://a/ uid=michael -LLL uid
 dn: uid=michael,ou=users,dc=michaelh,dc=hopto,dc=org
 uid: michael

 m@z ~ %

However, on a they don't:

 m@a(172.16.1.15) ~ % ldapsearch -x -H ldaps://a/ uid=michael -LLL uid
 ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
 m@a(172.16.1.15) ~ %

In the log (running slapd with -d 768), I see:

conn=20 fd=26 ACCEPT from IP=172.16.1.1:49298 (IP=172.16.1.1:636)
conn=20 fd=26 TLS established tls_ssf=16 ssf=16
conn=20 fd=26 closed (connection lost)

In fact, it appears that I was missing a key directive:

tls_cacert /etc/ssl/certs/MichaelSecurePlaces.pem

Having just added it to /etc/nss-ldapd.conf, I get a slightly worrying message:

 % sudo invoke-rc.d nslcd restart
Restarting nss-ldapd connection daemon: nslcdnslcd: /etc/nss-ldapd.conf:25: option tls_cacertfile is currently untested (please report any successes)

However, for now it seems to work...


Note: this is all running on a Debian lenny/sid system.
This is what happens when compressedchaos goes away for a week, and I start my holidays early.

The epic tale of how my new mail setup was born

My personal email has been on a rough ride over the years: from a reasonably nice (Microsoft based!) school email setup in 1999, which sadly got removed when Windows 98 was introduced, I went through three Hotmail accounts. I had a brief flirtation with GMail, but not being all that keen on the means of delivering advertising, I ended up back on Hotmail.

Registering dnorth.net last year at least ensured my email address would no longer change, but the technical capabilities of the two mail servers holding the mail still left much to be desired: flaky, unreliable spam scoring, no facilities for server-side filtering/sorting, sheer lack of customisability…

Last week, I finally did something about it. At my disposal was my VPS, running Debian Linux 4 (’etch’). On the wishlist were:

• Accurate server-side spam scoring with SMTP-time rejection of the most obvious spam
• Sender verification
• Sieve filters for server-side sorting into folders
• All mail stored on the server and accessible over secure IMAP

Thankfully, none of the above is too difficult: some pretty good instructions are Out There for most of it. The ones I used were:

• Exim4 on Debian with SpamAssassin, ClamAV, Virtual Domain Alias Files and Message Size Limits per Domain
• Exim + Sieve + Avelsieve Howto
• Handling mail for multiple virtual domains with exim4

Please remember, I am not responsible for the content of external sites (e.g., the links above), nor can I accept any responsibility for the consequences of acting on the points below…

I ran into a couple of issues:

• Permissions on the .sievesource files generated by AvelSieve - I needed to chmod g+w on /var/lib/squirrelmail/data and chown it to www-data:www-data in order to reach a state where Exim could read the file, and Avelsieve could write it.
• Exim4’s native sieve implementation only has the core features in it, not the extensions defined in RFC 3431. I dodged the issue by matching the number of *s in the X-Spam-Score header using string matching, rather than numeric checks on the X-Spam-Score.

All in all, though, it’s working a treat. Email perfection at last!