mass file storage question 1

[IDTstudios1]

Here's the situation:

There will be about 25,000 xml files that will range in size from 100kb to 5mb. Around 600 to 1,000 of these files will need to be accessed every minute.

Now, I have two options as far as storing these files:

1) MySQL for authentication and just save the files as whatever.xml in a neat little file structure.

Or

2) Since the files are in XML format I could break them apart and store them in the same (or different) MySQL database used for the authentication.


Which of the two would be my best bet here? Also keep in mind that there will need to be some load balancing between several servers.

Any input welcomed,
Andrew

 

[jemminger]

>> "Any input welcomed"
i like coffee ice cream.

-jeff

http://www.jeffemminger.com

try { succeed(); } catch(E) { tryAgain(); } finally { rtfm(); }
i like your sleeves...they're real big

 

[IDTstudios1]

well, I suppose that would be "input." Just to clarify for others, any input that is relevant to the post is welcomed.

btw, Birthday cake ice cream is much better...

 

[sleipnir214]

I have no hard data to support my decision, but I think I would store as much of the XML data as possible in some kind of hierarchical form in MySQL.


Want the best answers? Ask the best questions!

TANSTAAFL!!

 

[IDTstudios1]

That's what I was leaning towards also. What about speed? Which would be faster access a file or or having mysql re-create a file every time someone needs to download it?

Thanks

 

[sleipnir214]

Aahhh....

You didn't say what you were going to do with all this. I assumed users would access the data, not the files themselves.

If the users are going to download the files, you will get much better performance from leaving them on the filesystem. That way, PHP doesn't have to get involved in the streaming of the files -- your database server can handle it.


Want the best answers? Ask the best questions!

TANSTAAFL!!

 

[IDTstudios1]

Thanks for all the help sleipnir214.

 

[sleipnir214]

CORRECTION:

In my last post, the phrase "your database server can handle it." should have read "your [red]web[/red] server can handle it."


Want the best answers? Ask the best questions!

TANSTAAFL!!

 

[lgarner]

An orderly, nested directory structure might help the speed as well.

Something like PHP uses for session.save_path.

 

[sleipnir214]

The last time I checked, PHP's session save directory was organized as a flat set of files.


Want the best answers? Ask the best questions!

TANSTAAFL!!