Files Per Folder

[acepro]

Has anyone done tests to establish what the optimum
number of image files per winxp folder might be.

I have a catalog system with about 50,000 .jpg images
running on vfp9 winxp pro.

Right now I have it set up with a 6 digit numbering
system like 523640.jpg so the images are stored in
c:\image\52\36\523640.jpg. That means never more than
100 images per folder but lots of folders.File sizes
range from 10kb-50kb each.

Any thoughts on the tradeoffs with fewer larger folders?


Charlie Huff
CJ's Homecenter

 

[Olaf Doschke]

Besides the performance of the Windows Explorer, that slows down with many files per folder, I don't know issues.

I have done a flat folder structure which would compare to your structure if you had c:\image\5236\ for files 5236*.jpg and so forth, so only one level of folders. But opening files etc. is not depending on how much more files or folders are there in my experience. In my case it's not pictures, but tables and databases. And operations like open database or use table are as fast as in a disk volume with fewer files.

Besides other file viewers also have less performance issues with displaying such a huge folder/file structure, eg norton commander, total commander...

Otherwise:
NTFS: max 4,294,967,295 files per volume, nothing said about folder limits.
FAT32: max 4,177,920 files per volume , max 65,534 files per folder, which can be significantly reduced if files have long names.
FAT16: max 65,536 files per volume, max 512 files and folders within the root folder of a volume.

http://www.microsoft.com/resources/documentation/Windows/XP/all/reskit/en-us/prkc_fil_tdrn.asp

Bye, Olaf.

 

[Mike Lewis]

Charlie,

In general, the more files there are in a folder, the longer it will take Windows to find the one you want. When an application asks the OS for a file, the OS has to search the entire folder until it finds the required filename. Normally, this isn't a problem because the search times are pretty negligible, but if you have tens of thousands of names, it's might take a noticeable amount of time.

Once you've opened the file, any subsequent operations won't be affected by the size of the folder.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

My Visual FoxPro site: www.ml-consult.co.uk

 

[acepro]

Thanks Olaf,Mike

Sounds like my current structure which was set up
for ease of reviewing the images for updates is as
good as any.The path is short and straight forward
and 100 images in a folder not bad for the user to
view and replace.

Just hadn't ever worked with that many images before,
and wanted to reconfirm that my inclination towards
a short path name and short list at the end didn't
have some performance penalities.

Thanks for your reply

Charlie Huff
CJ's Homecenter

 

[craigsboyd]

Hi Charlie,

Depending on what your users are doing with the images, you could put the paths to the images in a table that is indexed and provide the users a screen for whatever manipulation they are doing. That way they would never have to touch Windows Explorer and finding images would be almost instantaneous. Also, the table would allow you to mark up the images with some metadata or if there is already metadata contained in the files as EXIF it could be read into the table at the time that the image paths are imported.

I'm assuming that these images are in some way used by or for a Visual FoxPro application. So I'm thinking there is a better way of organizing, searching, viewing, manipulating, and/or replacing the images than having your users use Windows Explorer.

What is it that the users are doing with the images? I guess I'm wondering what you mean by "view and replace" in your above post.

SweetPotato Software Website
My Blog

 

[acepro]

Craig

Sorry I was to focused on the file handling and didn't
give a more complete picture of the enviorment.

This is a retailer's catalog system with two main forms.
One is the detail view with a single sku,image,desc, etc.
The second is a page view with 12 sku's and only basic info.
The user can look up items using 8 different indexes
and view detail or pages in the sequence chosen.

The sku# is the file name for all stock images but there is
also a field in the main table for storing the file name of
a custom image if the user wishes to substitute their own.
This is accomplished by clicking on the image # in the
detail screen and directly entering the file name or rclick
which is the getfile() dialog.

The performance issue only relates to displaying 12 images
at a time on the page view when they may be on cd,local, or
on a server. The second issue is visually locating a custom
image to make it more user friendly. I advise the users to
use the same folder/file naming convention.

Hope that's a little clearer.

Thanks for the reply

Charlie Huff
CJ's Homecenter

 

[ChrisCarroll]

I suspect that an 'optimum' solution (whatever I mean by that) is several thousand files per folder because:

It used to be the case on Dos/Fat that you'd get a noticeable slowdown after a couple of hundred files. In other words, even on FAT you're okay with 100 files per folder.

NTFS wiped that slowdown effect out (by indexing directories, I think)

Ergo NTFS is probably optimized for hundreds to thousands of files.

(And, Windows SharePoint defaults to maximum 10,000 files per folder. Or something like that).


Chris

 

[GriffMG]

I *Think* you are failing to do the math fellas

There were limits on the root directory in FAT - in practice you were limited to 255 files in the FAT (less a few for the std devices).

I would expect there to be a similar limitation, based on a binary integer of some kind... 1024, 2048 or whatever.

So, even with indexing, keep the number to say 32,000 ish and you will do well enough - and you can rely on the indexing to 'cover your...' if you get it wrong!

B-)

Regards

Griff
Keep [Smile]ing

 

[Mike Lewis]

Griff,

I would expect there to be a similar limitation, based on a binary integer of some kind... 1024, 2048 or whatever.

Correct me if I'm wrong, but doesn't that limit only apply to the root directory?

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

My Visual FoxPro site: www.ml-consult.co.uk

 

[GriffMG]

Yes Mike

The root directory only

Martin

Regards

Griff
Keep [Smile]ing