Best dfs for this use...?

[overdie]

Hi All,
I've to choose a way of storing some media content of a social network. I'll have some photos (about500k each), videos(1-2MB each), audio (500k-1MB each) and user avatars. I'll have no metadata to associate so the use of a DB such mongoDB is not necessary.I'm thinking about a distributed filesystem like HDFS (but i've red about problems with small files) or AFS (all the case studies i've found are about people using it as user home, with a lot of clients while i'll have just the application acting as client).
Do you know what kind of filesystem are used by other social networks? do you have any clue to help me in the choose?
Thanks

 

[Annihilannic]

Many use "cloud" filesystems like Apache Hadoop. Also look at things like Lustre, Gluster (sp?), etc. You say "just the application"... does that mean just one client application and one server?

Annihilannic.

 

[overdie]

Hi Annihilannic,
thanks for your answer.
I've already inspected the Apache Hadoop but it is optimized for large files, i'll have just little files (max size will be 3 MB i think).
with "just the application" i meant that i'll have just the application servers (starting with one, growing in number with the amount of users i'll have) and not a lot of clients asking for their home directory (that's the major use i've found for AFS, for example).
I've looked at Gluster, it seemed quite complex, i wouldn't make the things more difficult than they could be.
Do you have any notice about Lustre and its use by some social?

 

[BadBigBen]

my personal favorite would have to be XFS, it's pretty good on performance and reliability, though better suited for large files it never the less does darn well with small files...

alternatively, you could also just stick with EXT4, it also is suited for the listed needs...

and frankly, you do not need a clustered filesystem, unless you are talking about a SAN environment...

the following has some insights, though a bit dated (2006):

Filesystems (ext3, reiser, xfs, jfs) comparison on Debian Etch

http://www.debian-administration.org/articles/388

Ben
"If it works don't fix it! If it doesn't use a sledgehammer..."
How to ask a question, when posting them to a professional forum.
Only ask questions with yes/no answers if you want "yes" or "no"

 

[Annihilannic]

BBB, the question was about a dfs... none of those you listed qualify?

overdie, it does depend quite a lot on your intended storage architecture... is it going to be using shared storage like BBB mentioned? In which case I'd try something like Red Hat's GFS.

There's a good list of distributed and shared filesystems on Wikipedia.

Annihilannic.

 

[BadBigBen]

Correct Annihilannic, in that the ones I listed are not DFS types...

that was one of the classic "Type before thinking" type deals, on my part...

but still, from the data he listed, I would conjecture that a DFS would be overkill, and would just add to the complexity of the whole setup...

plain old NFS would fit the bill there, and is implemented already in most (if not all) Linux distros...


Ben
"If it works don't fix it! If it doesn't use a sledgehammer..."
How to ask a question, when posting them to a professional forum.
Only ask questions with yes/no answers if you want "yes" or "no"

 

[overdie]

Thank you guys for your answers...
actually, like BBB said, the choice is between the complexity of a DFS (AFS,LUSTRE,...) where, after set up, there's no need to take care of where and how files are stored or NFS (i need a quite scalable storage solution and i'm scared about adding space or hosts with GFS while in use) with the duty to think in the application a way of storing and searching for files..
Additionally with NFS i'll be able to tune each mount for different kinds of files, i don't know if it's possible with the actual DFSes mentioned before...do you have any notice about the way of storing files used by social networks?

Thanks!

 

[rzs0502]

Just a though - how about using FreeNAS for your storage.
It now includes ZFS support so you'll have good performace with snapshots etc.
Other clients can connect to the storage irrespective of OS.
So you can have a mixture of OSes using the storage.

Clustered filesystems like coda and ocfs can more problematic.
If you go this route, lustre is great as someone has mentioned.
It has been used in many a super computer. That's why Oracle has snatched it.



"If you always do what you've always done, you will always be where you've always been."