If I recall (and it's a big if), the directory size grows due to the inodes which are associated with it, so deleting files doesn't have any effect. Possibly the only way to return it to 'normal' is to delete it and recreate it, though whether you'd want to do this is debateable. Someone else might well have a more definitive answer. Cheers.