After you sign up, you’re listed in the current auctions. People can come along and bid for a date with you. You are in complete control over who you accept. No need to worry about some weird (but rich) person winning your auction. You can pick the person you most like the look of (if they show suitable generosity to your chosen charity of course!)

Despite being brand new, there have already been a number of bidding wars over some people, reaching into the hundreds of pounds.

You can read more about Donate 2 Date here.

I was initially skeptical about having ads. I hate radio stations that will have you listening to adverts all day long or TV stations that play a few minutes of ads every 20 minutes. But adverts in Spotify are rare enough that it doesn’t bother me. If you don’t like the adverts, you can pay a subscription at £9.99 a month or a day pass for £0.99 (for parties perhaps?). I think the £9.99 a month is a little steep, but luckily I’m happy with the ad-supported version.

They have most of the music I like to listen to. However, they often don’t have the obscure first albums (produced by tiny labels), which is a real shame. I expect this will change if Spotify gets big. There are a couple of big names who are not available. The only one I have really missed is Metallica’s new album.

Each artist has links to similar artists. This is nice for finding new music, but doesn’t appear to be particularly clever. That said, exploring is effortless because songs are played instantly and the browsing interface is very well designed. Songs and playlists can be sent to friends using special URLs, but these have to be emailed or IMed, it would be much better to have a more tightly integrated social network. I’m sure this will follow later on.

At the moment, Spotify is invitation only. I have a few spare, leave me your email and I’ll send you one if I have one left.

EC2 was one of the first and most mature of the offerings. Though it does not stray far from the traditional concept of a machine (with fixed CPU and memory), what it does do is give you access to as many of these as you need at any one point and the (very) basic tools to manage them. They provide the basic infrastructure: machines, network connections and data storage. It is up to you to do the rest: load balancing, application servers, relational databases, backups, replication, redundancy, fail-over, etc.

Scalr promises to help with some of these issues, notably load balancing, and backups / replication / fail-over for MySQL. The idea is great but the project is not mature (despite the claim to v1.0) and the architecture is not very solid. To be fair, this is acknowledged to some degree in that v2 is to be a complete redesign. After a period of two months using Scalr, I decided to move on because of these issues:

• Ad-hoc design: a strange combination of PHP, bash, MySQL and CRON. The interaction between these components was overly complicated, which means debugging was tricky and the system itself was prone to getting mixed up (for example: CRON calling PHP to do a backup but the MySQL state was stuck so backups were being skipped, silently, for weeks). V2 is to be Java.
• Buggy: rebundling an instance would break the load balancer. It would get confused about the new instances and refuse to update. This was only solved by restarting the whole farm.
• Scalr makes the decision to start new instances based on the load average. This is over simplistic really and can mean that your machines are under heavy load for a little too long which new instances are being brought up.
• Scalr is built on Ubuntu 7.04 AMIs, upgrading the distro breaks it.

My first reaction was that it was some sort of virus or at best a rogue app, but googling mdworker revealed that it was part of OS X. More precisely it is the Spotlight indexing daemon (keeps track of all your files to make searching faster).

My first reaction was to look for a way to turn off spotlight. I don’t really use it very much and Quicksilver covers the needs I do have. However, Spotlight is also needed to search your mail, which I can’t do without.

The problem appears when Spotlight is trying to keep up with a large number of new files on the system. In my case copying some music onto a new machine. The most straightforward solution is to tell Spotlight to ignore those files. This is easily done in System Preferences -> Spotlight -> Privacy -> Add ( + ). I added my Music directory which is where iTunes stores it’s library.

It is odd that this should be a problem. Indexing is something that should happen in the background. There is no reason for it to affect the user. It appears that Apple decided that it was more important for files to be searchable than for the operating system to be responsive.

I’m currently building a Facebook app using Scalr to deal with scaling. This automatically creates new Amazon EC2 instances to deal with increases in load and kills them again when traffic dies down. The problem arises in deciding when to start a new instance.

Scalr keeps track of the server load average.  This is a moving average over 15 minutes. It’s a good idea to use a slow moving average like this so as to not start instances prematurely. It will only do so if the existing instances do actually need some help (especially considering Amazon charge by the hour). However, this introduces lag in starting a new server, say about 5 minutes.

If I weren’t serving pages to users via Facebook this wouldn’t be such a problem. There would be small window when the site would be a little sluggish. For a canvas page this means pages are considered to have timed out, not good!

Of course, using IFrame’s is generally faster because it doesn’t have the proxy overhead (and sometimes sluggish Facebook servers). There are some drawbacks in losing some cool FBML functionality (e.g. fb:can-see) and you will have to make a lot more calls to the API, but I now think IFrames are the way to go.

The most recent version is 0.4, so it obviously does not claim to be ‘finished’ (can software ever be?) However, there are plenty of benefits in using it already:

• Ridding your beautiful Python code of any SQL. It’s untidy and pain to work with (which means bugs).
• ORM (Object Relational Model) to fit in with a nice OO architecture.
• Free sharding to split your database horizontally.
• DB independant code, so you can switch if you feel the need.
• Hassle free transactions and DB connection management.

Sounds great doesn’t it? In general this is exactly what you get. However, there are a few points which are very important to consider before committing yourself to using SQLA.

Forcing Indices

Anyone who has had to do something reasonably involved in MySQL has spent time optimising their indices. Sadly, MySQL is not very clever about how it chooses the index to use for a particular query. In some cases the only option is to force it with FORCE.

This is not available in SQLA, so anytime you need to do this, you’ll have to manually specify the SQL. Not only does this partly defeat the point of SQLA, but it also happens that the place you most need the SQL abstraction is often the same place you need to force an index. Consider a complicated search page. You are searching over a large data set using a number of filters. MySQL is probably going to get the index wrong and you’re going to have to generate a complicated SQL query without SQLAlchemy’s help.

Commit ORM Objects

SQLAlchemy can be a little silly about detecting a change in an ORM object. If you assign to a member variable which is part of the schema, it will be marked as dirty even if the value doesn’t actually change.

The solution is to check for a change before assigning, which does not make for neat code!

Sharding

It’s a good idea to plan for scaling as early as possible. It can be very difficult to build the necessary bits in later when you need it (and will also under pressure to fix things fast). One common way to deal with scale is to split large tables over several machines, this is known as sharding (each independant chunk of table is known as a shard).

SQLAlchemy has some code to help you there. All you need to do is to define 3 functions which tell SQLA which shard a particular row is in. Really simple. Sadly, this part of the code is not very mature at all:

• query.count() doesn’t work (nor any scalar query). I had to write a function to query each shard in turn and sum the result. The real problem here is that it was not clear at all which bits of functionality will or won’t work with sharding (expect long debugging sessions, digging into the SQLA code).
• The ORM caches objects and identifies them by their primary key. However, a common MySQL trick when sharding is to have an auto_incremented INT as the primary key of each shard but use something like a UUID as the ‘real’ primary key recognised by the code (this speeds things up quite a bit). Of course, the auto_incremented key will not be unique across shards and this will confuse SQLAlchemy. I think the best solution here (suggested by someone on the SQLAlchemy group) is to have a 2 column primary key with an INT and another integer shard identifier, making it unique.

No Server Side Cursors

A minor point, but might be important for some. Server side cursors come in handy sometimes when you are dealing with large amounts of data.

Tips For Optimising

• Periodically check the SQL queries being made with the echo option. There might be some surprises in there (though usually easily fixed). This kind of thing usually pops up because you will use the ORM instance and forget / not realise that it will result in a query (from a software design point of view, great. From an optimisation point of view, awful).
• Use set_shard on a query whenever you are able. If you know which shard the column you want is, no need to go checking the other ones. A common example is when the shard identitifier is in one of the query parameters.
• Design for scale right from the beginning and develop / test on a distributed architecture (i.e. have at least 2 shards). This doesn’t need to be difficult, for example: just create two databases on your developement box to simulate two machines.

Conclusion

I still think that SQLAlchemy is worth using. Overall it will save time and effort as long as you are careful and not afraid to get your hands dirty when the going gets tough. I expect most of these problems will be addressed in (near) future releases.

It was a couple of years before I started to use it again. In the meantime I'd mostly been working with .NET and Java so I'd been using some pretty decent development environments and it didn't seem necessary to use anything else (especially considering the power of their debuggers). But I moved into the world of the web and started writing PHP and then Python, these did not really have especially good IDEs and so it was back to a text editor and my choice was Vim. It also coincided with worsening RSI, for which Vim is great.

I tend to learn a couple of features in a burst every few months when something really bugs me. This is probably not ideal, but Vim is so powerful I will never learn it all. I've been keeping a to do list in a text file recently (rather than on paper - go planet!) but Vim's defaults are not great for editing prose:

• Vim's word wrap is by character, not word.
• k and j (up and down) work on a line basis. If you have a wrapped line, you cannot move inside it with j and k (like you would with a normal text editor).

Of course, Vim is hugely powerful and can be tweaked to be much more useable when writing prose. These .vimrc commands:

• Wrap lines by cutting lines off at word boundaries.
• The word wrap is virtual, no extra line break is inserted (so that it's easy to edit afterwards).
• j and k are replaced with gj and gk which allow you to move up and down inside a wrapped line.
• I've also added the spell checker (I've not played around with it properly yet, but it looks a little weak).
• smartindent for bullet points.

On a slightly different note: if you're using OS X, get this port of Vim. It's aim (and it does) is to integrate better into Macs. It's worth getting just because it has pretty Carbon tabs

It uses the built-in Firefox alert system, so you'll need Firefox 3 on OS X (for Growl) and 2 for other OSs. At the moment you also have to leave the Facebook page open, if I have time later, I'll deal with this so that you can close the Facebook page.

PLUG: Donate 2 Date is my Facebook app. It helps raise money for charity by setting people up on dates. Check it out!