I would attack the issue like this:
When ever an image needs to be stored create a surrogate key consisting of the logged in user name + the image name e.g. fredJennyonthelawn.jpg
Create a hash of this key.
Need to stop here, assuming we have 10 web servers the images need to be spread across these 10 servers. So we have to map the hash value across the 10 servers which we might do by taking the modulus of the hash and 10 so we end up with a number between 1 and 10 which we will call server number. We can allocate the image to that server and when we want to get it back do the process in reverse so we identify the server and then issue a call to retrieve it. The obvious way would be via the web server as this takes away the connectivity issue.
But we are left with a problem. When we mod by 10 all is well is good. If we add another server we would then have to mod by 11 which will give us different results to when we use 10. For example if the hash of fredJennyonthelawn.jpg is 38 moding this with 10 gives us 8 but mod 11 gives us 5, in effect our images would be lost. We need to maintain the server number regardless of which physical server the image is stored on.
To get around this it is common to use some indirection or a logical to physical map. You would choose a value (say 50) which will represent the maximum number of servers you would ever scale to so our server number will never change as this is the value we use for the mod calculation. This should be a high value as making it too small will require all the hash values to be recalculated and balanced and being a high number is not an issue.
To illustrate I'll use smaller values of 4 physical servers and logical servers of 12. At a first pass the logical to physical map would be:
logical: 1 2 3 4 5 6 7 8 9 10 11 12
physical: 1 2 3 4 1 2 3 4 1 2 3 4
So logical server 7 would map to physical server 3.
If we now had to add a new physical server to give us 5 our map would look like.
logical: 1 2 3 4 5 6 7 8 9 10 11 12
physical: 1 2 3 4 5 1 2 3 4 5 1 2
So our logical server 7 now is actually physical server 2.
We now have another problem as we the images are now on the wrong server!. This is not a really big issue as introducing the new server probably meant that we were running out of space any way so we should the opportunity to rebalance in any case. So what we have achieved is we can store a scalable number of images and always know where they are, the application doesn't know or care as it only sees the logical server.
Of course the indirection an be done in a database with a pointer. This is just an alternative and gives you some transparent method of method of balancing images to servers (which is where the technique comes from, sharding)
