Too Many Files Open using FindNextFile and FindFirstFile

[areric]

Hey all,

Sorry for the low points on this, kinda used most of them up. Hopefully someone will still have the answer.

Anyway im trying to run a recursive file scan of a file server. The Server has about 3TB of data so its pretty big. Anyway im using Windows API calls and using FindFirstFile and FindNextFile.

The scanner works perfectly on a small test folder, but when i get to a larger folder i run into issues.

The scanner will enter and leave directories recussively for a good while, scanning about 70,000 files and directories. However after a while it eventually dies. I added in some calls to GetLastError and some logging and found out that The function tried to make a call to GetFirstFile of a directory and returns an error code of 4. That equates out to being "Too many open files". Does anyone know how i can get around this problem?

My code is pasted below for the function having the issues, thanks.

bool FindFile::getFiles (HANDLE &searchHandle, WIN32_FIND_DATA &fileData,
string path)
{
int nValid;

if (searchHandle == NULL)
{
string pathToSearch = combinePath(path, "*");

searchHandle = FindFirstFile(pathToSearch.c_str(), &fileData);
if (searchHandle == INVALID_HANDLE_VALUE)
{
cout<<"FindFirstFile"<<endl;
cout<<GetLastError();
if (GetLastError() == ERROR_NO_MORE_FILES)
{
FindClose(searchHandle);
searchHandle = NULL;
}

}
nValid = (searchHandle == INVALID_HANDLE_VALUE) ? 0 : 1;
}
else
{
nValid = FindNextFile(searchHandle, &fileData);
if (nValid == 0) {
cout<<GetLastError();
if (GetLastError() == ERROR_NO_MORE_FILES)
{
FindClose(searchHandle);
searchHandle = NULL;
}
}

}

while (nValid)
{
// As long as this file is not . or .., we are done
if (strcmpi (fileData.cFileName, ".") != 0 &&
strcmpi (fileData.cFileName, "..") != 0)
return true;

nValid = FindNextFile(searchHandle, &fileData);
if (nValid == 0) {
cout<<GetLastError();
if (GetLastError() == ERROR_NO_MORE_FILES)
{
FindClose(searchHandle);
searchHandle = NULL;
}
}
}

FindClose(searchHandle);
searchHandle = NULL;

return false;
}

p.s. if the code looks firmilar some of it (before my modifications) was taken from a program by Louka Dlagnekov that I found online. Just dont want people to think im claiming it as my own.

 

[Salem]

Like all resource leak problems, the code which spots the problem is seldom the code which causes the problem.

Have a look at the code which calls this function, to see if its leaving any handles unreleased.

I can see that your final close is equivalent to
FindClose(NULL);

Can you for instance do

dir /b/s

on the directory tree?

> The Server has about 3TB of data
But how big is the directory structure?
How deeply nested is it?

--

 

[ArkM]

Yet another suspicious points: where is recursive mechanics and where is DIRECTORY attribute checking?..
I think directory tree height is not so large to raise stack problems in a proper recursive directory scanning...

 

[areric]

The directory tree is highly nested. It is the global team storage for a fairly large company so at points it could go 20-30 folders deep.

At the root theres only about 40 folders but they represent the different units of the company and from there it divides into divisions, teams, etc.

Here is the code that makes the recursive calls
int FindFile::scanPath(string path)
{
WIN32_FIND_DATA fileData;
FileInformation fi;
int success=0;
char* acctName;
string filename;
int sum = 0;
int filesum = 0;
_ConnectionPtr conn;

HANDLE searchHandle = NULL;

if (m_opts.extendedInsert == true) {

DbConnect(conn);
}

while (getFiles (searchHandle, fileData, path))
{
// Skip this file/directory if not matching criteria
/*if (!matchCriteria(fileData))
continue;
*/

//If this file is hidden skip it.
if (fileData.dwFileAttributes & FILE_ATTRIBUTE_HIDDEN)
continue;
//If this is a directory, scan it and add its files to the sum
if (m_opts.recursive && fileData.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY){
if (path == m_opts.location)
{
m_opts.shareName = fileData.cFileName;
}
m_opts.currentLoc = fileData.cFileName;
filesum = filesum + scanPath (combinePath (path, fileData.cFileName));
//Insert directory Record into DB.
if (m_opts.returnFolders){
fi.fileinfo = fileData;
fi.path = path;
fi.filesize = filesum;
filename = combinePath(fi.path, fi.fileinfo.cFileName);
success = getOwnerName(filename.c_str(), acctName);
if (isQLid(acctName))
{
fi.qlID = acctName;
}
else if(isQLid(m_opts.shareName))
{
fi.qlID = m_opts.shareName;
}
else
{
fi.qlID=acctName;
}
fi.location = m_opts.location;
fi.shareName = m_opts.shareName;
fi.isDirectory='y';
if (m_opts.extendedInsert == true){
InsertToDB(fi, conn);
}
else
{
filelist.push_back (fi);
}
}
}
else
{
filesum += ((fileData.nFileSizeLow + fileData.nFileSizeHigh * MAXDWORD)/1000000);
listsize += ((fileData.nFileSizeLow + fileData.nFileSizeHigh * MAXDWORD)/1000000);
if (m_opts.returnFiles){
fi.fileinfo = fileData;
fi.path = path;
fi.filesize = ((fileData.nFileSizeLow + fileData.nFileSizeHigh * MAXDWORD)/1000000);
filename = combinePath(fi.path, fi.fileinfo.cFileName);
success = getOwnerName(filename.c_str(), acctName);
if (isQLid(acctName))
{
fi.qlID = acctName;
}
else if(isQLid(m_opts.shareName))
{
fi.qlID = m_opts.shareName;
}
else
{
fi.qlID=acctName;
}
fi.location = m_opts.location;
fi.shareName = m_opts.shareName;
fi.isDirectory='n';
if (m_opts.extendedInsert == true){ InsertToDB(fi, conn);
}
else
{
filelist.push_back (fi);
}
}
}
sum += filesum;
filesum = 0;
}
return sum;
}

 

[PerFnurt]

I've never really got recursive findfirst/findnext calls to work. When traversing dir structures I always make sure I finish handling the current dir before recursing down.

Pseudo code:

void Traverse(string path)
{
  vector<string> dirs;
  FindFirst Bla bla bla
  while(FindNext)
  {
    if (isDirectory)
      dirs.push_back(theDirName);
  } 
  FindClose

  For each dir in dirs
    Traverse(path+"\"+dir)
}

/Per
[sub]
&quot;It was a work of art, flawless, sublime. A triumph equaled only by its monumental failure.&quot;[/sub]