Filer class has a bug or I'm missing something 2

[dkean4]

I have been massaging this code all weekend and need some help.

oFiler = createobject('Filer.FileUtil')
oFiler.SearchPath = "F:\"
oFiler.FileExpression = "*.*"
oFiler.SubFolder = 0
oFiler.Find(0)

for ix = 1 to oFiler.Files.Count
  ? oFiler.Files(m.ix).Path, oFiler.Files(m.ix).Name
endfor

Works perfect! And it is quick. But it only works on the particular directory specified in oFiler.SearchPath. In this case the Search Path is the drive "F:\". The reason it works only for the immediate Path to the folder is because oFiler.Subfolder is set to "0". Works flawlessly.

So, let's change up oFiler.Subfolder to "1" and try it again:

oFiler = createobject('Filer.FileUtil')
oFiler.SearchPath = "F:\"
oFiler.FileExpression = "*.*"
oFiler.SubFolder = 1
oFiler.Find(0)

for ix = 1 to oFiler.Files.Count
  ? oFiler.Files(m.ix).Path, oFiler.Files(m.ix).Name
endfor

Well, now when I run it, and I tried it on 2 different WIN7 machines, it appears to lock up. But it is not locked up. It waits for the user to press ESC and subsequently runs the FOR loop. And it is ready to run the loop as soon as the user clicks the key. The speed of processing the entire list is impressive. For me that is a winner! I could not ask for more.

The only problem is "Why does it turn the cursor into a 'Processing' icon and requires for the user to press the ESCAPE key. What is it doing? What am I missing? Is there another switch that needs to be set to continue when the Subfolders switch is set to on or 1? I am baffled! Many people refer to this short script, on the WEB, and I assume it must work for them.

TIA

Dennis

 

[atlopes]

It seems that Filer.dll has its own ESC trapping mechanism.

oFiler = createobject('Filer.FileUtil')
oFiler.SearchPath = "F:"
oFiler.FileExpression = "*.*"
oFiler.SubFolder = 1
SET ESCAPE OFF
WAIT WINDOW "Working..." NOWAIT
m.nEscaped = oFiler.Find(0)
WAIT CLEAR
SET ESCAPE ON

if m.nEscaped = -1
  messagebox("interrrupted")
endif

for ix = 1 to oFiler.Files.Count
  ? oFiler.Files(m.ix).Path, oFiler.Files(m.ix).Name
endfor

 

[Olaf Doschke]

From what you and antlopoes say it seems the whole find mechanism can be interrupted/ended via ESC. If you have early found files it may seem fast to find files, but you don't have the full drives files, if you ESC and don't wait. How large is drive F:\ In the range of 100GBs or even TBs? Scanning a full file system takes time, you get lots of files right aways, so the result looks fine, if you ESC early, but you seem to think File is finished already, I don't think so.

Bye, Olaf.

 

[Mike Lewis]

Hi Dennis,

I've just been playing with this code. You're right - it is very fast.

I'm sorry I can't offer any suggestions for dealing with the ESC problem. But I did notice that you don't have to hit ESC to get past the hourglass. In fact, you can press any key. I know that doesn't help. But I just thought I'd mention it, for the same of completeness.

I did wonder about inserting [tt]KEYBOARD CHR(27)[/tt] into the code, but that appears to terminate the processing immediately.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[dkean4]

Atlopes,

Does it work for you, or do you get the same effect? I ran your code, but it is doing the same thing. If this would work there would be no need for recursion. And it is lightning fast.

Simplicity is the extreme degree of sophistication.
Leonardo da Vinci

 

[dkean4]

Olaf, pressing ESC delivers the whole load. I tested it more than once in total disbelief. It grabs every file no matter how many there are. For my project it is a dream. Escaping does not abort! Thanks for the heads up anyway.


Dennis

Simplicity is the extreme degree of sophistication.
Leonardo da Vinci

 

[atlopes]

When the result set is small, .Find() returns immediately, even when .SubFolders = 1. As going through a large tree can be very time consuming, I believe that an internal ESCape trapping was incorporated to avoid what could be interpreted as a machine lockup.

This is not documented, but that's the way the companion Filer.SCX works: SETs ESCAPE OFF, then launchs the .Find() method, then restarts the ESCAPE setting. .Find() returns either the number of file and folders (this in the help file) or -1, in case the user pressed ESC (this can be assumed from the Filer.SCX code). Therefore, if you can list your drive contents using Filer.SCX, you should also be able to do it by calling yourself the .Find() method.

This code goes through a tree with 17478 entries in 19 seconds, no need to press ESC to execute:

oFiler = createobject('Filer.FileUtil')
oFiler.SearchPath = GETDIR()
oFiler.FileExpression = "*.*"
oFiler.SubFolder = 1
SET ESCAPE OFF
WAIT WINDOW "Working..." NOWAIT
m.Start = DATETIME()
m.Escaped = m.oFiler.Find(0) = -1
m.Lapsed = DATETIME() - m.Start
WAIT CLEAR
SET ESCAPE ON

if m.Escaped
  messagebox("interrrupted")
endif

MESSAGEBOX("Read " + TRANSFORM(m.oFiler.Files.Count) + " entries in " + TRANSFORM(m.Lapsed) + " seconds")

for ix = 1 to oFiler.Files.Count
  ? oFiler.Files(m.ix).Path, oFiler.Files(m.ix).Name
endfor

 

[atlopes]

Dennis, I tested with a whole drive and you're right, somehow at this scale it fails to return the result unless you press ESC. But the result is there in the collection.

Must go to attend classes, now, but will look at this later on...

 

[dkean4]

Mike, I tried my own tricks by creating a timer of 1 second on the fly with CREATEOBJECT() and letting it fire a KEYBOARD ESC key. The timer, for some reason will not fire until the user presses the ESC or if you are right, any key.

CLEAR

SET CLASSLIB TO .\classes\phook ADDITIVE 
wait_killer = CREATEOBJECT("escaper")
wait_killer.enabled = .T.
 
oFiler = createobject('Filer.FileUtil')
oFiler.SearchPath = "G:\"
oFiler.FileExpression = "*.*"
oFiler.SubFolder = 1
oFiler.Find(0)

for ix = 1 to oFiler.Files.Count
  ? oFiler.Files(m.ix).Path, oFiler.Files(m.ix).Name
ENDFOR

This is the simple code to send an ESC key with the timer:

This.Enabled =.F.
@ 20,10 say "SUCCESS!"
KEYBOARD '{ESC}'

But it is suspended and resumes the the timeout after I press the first ESC key manually... I guess MS does not want us to have this precious resource... What else can I conclude...


Simplicity is the extreme degree of sophistication.
Leonardo da Vinci

 

[Mike Lewis]

I guess MS does not want us to have this precious resource..

More likely a bug or design fault (in Filer.DLL). If they didn't want us to have it, they wouldn't have given us the sample.

But you're right. It would be a pity not to be able to use it.

For what it's worth, I use my own hand-coded file searching routine, based on recursive calls to ADIR(). Not surprisingly, it's slow. Even without any progress bar or other display to slow it down, it takes about a minute to check 100,000 files. You could speed it up slightly by skipping things like the recycle bin and the temp folder, but that won't make a big difference.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[dkean4]

Atlopes,

My C:\ drive contains about 1,300,000 items including files and directories. And it takes it as little time as my human reflex to press ENTER to run and follow it up with my left hand to press ESC. And the entire collection of 1.3 million items is available to the FOR loop. I am stunned and thrilled and furious at MS for holding back. Same problem with giving us bum ComboBox and Grid classes which are the heartbeat of UI.

My PC is NOT a game PC. But I do have a 0.5TB SSD Samsung drive in it 8GB main memory and i5 quad core processor. It is decent, but not impressive. Clearly Filer is well designed and probably uses either the Index or a ready made directory collection by just passing it along in oFiler.Files. So, this was available all along.



Simplicity is the extreme degree of sophistication.
Leonardo da Vinci

 

[dkean4]

Mike,

Thank you for thinking with me on this, so far. It is so appreciated. I fear that I will have to design some external thread, lock to the window which invokes Filer and try to do a KEYPRESS ESC from the outside. I am naturally hard headed or persistent and presently hell bent on getting this accomplished. 1.3 Million items in the blink of an eye is a dream come true. I don't think that there is any fetching going on, by the way. As I said in the former post this collection must be sitting pretty and ready for retransmission and Filer passes it on in the Filer.Files property. What would be nice is finding the location of that permanent directory collection. We would not even need Filer. If you have any ideas, let me know.

By the way, is the source code for Filer.dll available on the WEB?

Again thanks for your suggestions, Mike.


Dennis

Simplicity is the extreme degree of sophistication.
Leonardo da Vinci

 

[Mike Lewis]

is the source code for Filer.dll available on the WEB?

I was wondering that as well. I can only say that if it is available, it is well hidden. Google doesn't seem to know about it.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Olaf Doschke]

I have index services off, at least I am sure for a certain drive (H and no, oFiler.Files is not the count of files, the count grows, the longer I wait, so this is just impressively fast, but not immediately returning the full set of files. You only get the full collection of all files, when you wait.

I have a very small drive set up with just 70 files and that's instantly in oFiler.Files, but not so on drive H:, where the longer I wait, the higher the file count.

You don't need to ESC, but if you do, you interrupt the build up of the Files collection. When filer gets through with all files (eg in my 70 files drive) the Find(0) call ends without the need to ESC.

Bye, Olaf.

 

[dkean4]

Olaf,

What size is your FILER.DLL file. I just downloaded a new one and it behaves the way yours does. But this file is half the size of the one that was in my PC before.

So, we are not on the same platform, then.


Dennis


Simplicity is the extreme degree of sophistication.
Leonardo da Vinci

 

[dkean4]

Olaf,

Regretfully, that suspension does not expire. It never completes. But on my PC it does accomplish the feat very fast and stays spinning its wheels. That speed is probably due to the SSD drive. Spindle drives are much much slower.



Dennis


Simplicity is the extreme degree of sophistication.
Leonardo da Vinci

 

[atlopes]

Additional info: when I try filer.dll (from filer.scx) from the root of my drive D:\, it never returns if I don't ESCape the search. And, no matter if I wait a long period of time or just a small amount, it always returns the same number of file names. I wonder if it gets stuck somewhere along the process, probably due to a security blocking it encounters that, because of its old age, can not solve properly.

So, I decided to create a class to traverse a tree of folders. It holds 3 collections: Tree (that is, the list of folders), Files (er... yeah, file names) and Blocked (the folders that, for some reason, could not be read).

In terms of speed, it seems faster than filer.dll (but, of course, it is not grabbing all the info a filer object supports, so it's not a fair race, if a race at all): my test came down from 18~19 to 3~4 seconds for the same tree. But, since we are having different experiences around here, that has yet to be confirmed.

It uses the File System Object, no ADIR().

CLEAR

LOCAL MyDisk AS FolderTree
LOCAL RootFolder AS String

LOCAL Start AS Datetime
LOCAL Lapsed AS Integer

m.MyDisk = CREATEOBJECT("FolderTree")

m.RootFolder = GETDIR()

m.Start = DATETIME()
m.MyDisk.Populate(m.RootFolder)
m.Lapsed = DATETIME() - m.Start

? m.MyDisk.Tree.Count, m.MyDisk.Files.Count, m.MyDisk.Blocked.Count
? "Traversed in",m.Lapsed,"seconds"

DEFINE CLASS FolderTree AS Custom

	FilerSystem = .NULL.
	Tree = .NULL.
	Files = .NULL.
	Blocked = .NULL.

	FUNCTION Init

		This.Tree = CREATEOBJECT("Collection")
		This.Files = CREATEOBJECT("Collection")
		This.Blocked = CREATEOBJECT("Collection")
		This.FilerSystem = CREATEOBJECT("Scripting.FileSystemObject")
		RETURN TYPE("This.FilerSystem") = "O" AND !ISNULL(This.FilerSystem)

	ENDFUNC

	FUNCTION Destroy

		This.Tree = .NULL.
		This.Files = .NULL.
		This.Blocked = .NULL.
		This.FilerSystem = .NULL.

	ENDFUNC

	FUNCTION Populate
	LPARAMETERS Root AS String
	
		This.Tree.Remove(-1)
		This.Files.Remove(-1)
		This.Blocked.Remove(-1)
		This._populate(m.Root, .NULL.)

	ENDFUNC

	HIDDEN FUNCTION _populate
	LPARAMETERS FolderName AS String, FolderObject AS Object
	LOCAL Path AS String
	LOCAL File AS Object
	LOCAL SubFolder AS Object

		IF ISNULL(m.FolderObject)
			m.FolderObject = This.FilerSystem.GetFolder(m.FolderName)
		ENDIF

		IF !ISNULL(m.FolderObject)

			m.Path = ADDBS(m.FolderObject.Path)
			This.Tree.Add(m.Path)

			TRY
				FOR EACH m.File IN m.FolderObject.Files
					This.Files.Add(m.Path + m.File.Name)
				ENDFOR

				FOR EACH m.SubFolder IN m.FolderObject.SubFolders
					This._populate(m.Path + m.SubFolder.Name, m.SubFolder)
				ENDFOR
			CATCH
				This.Blocked.Add(m.FolderName)
			ENDTRY

		ENDIF
	ENDFUNC

ENDDEFINE

 

[Olaf Doschke]

Filer comes with VFP and I used Version 9.0.0.2412.
A never ending filer may speak for some symlink causing a backloop, have you simply tried using a start folder of which you know it's full content including subfolders? Then it must end.

CLEAR

lcTemp = ADDBS(GETENV("TEMP"))
MKDIR (lcTemp+"somedir")
CD (lcTemp+"somedir")
STRTOFILE("hello","readme.txt")
MKDIR subdir
CD subdir
STRTOFILE("world","readme2.txt")

 
oFiler = createobject('Filer.FileUtil')
oFiler.SearchPath = lcTemp+"somedir"
oFiler.FileExpression = "*.*"
oFiler.SubFolder = 1
oFiler.Find(0)

for ix = 1 to oFiler.Files.Count
  ? oFiler.Files(m.ix).Path, oFiler.Files(m.ix).Name
ENDFOR

If this doesn't end for you, I don't know what's wrong, but it is no default behaviour of filer to wait for a key after finishing.

Bye, Olaf.

 

[dkean4]

atlopes,

Very impressive job, my friend.

I used your new construct, above, a few minutes ago and here are the results:
It takes 20 seconds to traverse my F:\ drive. It finds 63,739 files and 4,103 subdirectories nearly 67,000 items. You are accessing the file system directly through "Scripting.FileSystemObject". I am very impressed with that I was looking for that sort of OS access into various lists, data etc. Years ago I messed with this... to access device list and other and then promptly forgot it all. So, I must thank you for refreshing me with that. I need to find as many of these collections to facilitate some of the things I need to do. If you feel inclined point out how to access the device list and some other ones, I will be grateful.

So, here is where I stand, presently, atlopes. I did lots of testing this morning...
When I use Filer on my F:\ drive, I get files and directories loaded in 0.7seconds. There are 103,000 items in the Drive. And that was fooling me for a while that it was directly fetching and accessing everything instantly. I was wrong. What I intend to do next is to run Filer.DLL to get the whole drive list of items and save them in a DB. Then I can access any individual file within 2 seconds max...

Your example above is just what the doctor ordered, so though it is not as fast, it has a place in my arsenal. And I am very appreciative for your effort and the examples you put together, Atlopes.

With great regard for your help.


Dennis


Simplicity is the extreme degree of sophistication.
Leonardo da Vinci

 

[dkean4]

Olaf,

Good News... your construct does work! I guess that one or more of the branches are busted, in my C:\ directory structure. I will do a search for it by testing each 1st level subdirectory. Of course, it could be the sheer size of the file system, but the task is still within the realm of feasibility by parts. I tested the shortest time it takes to complete 350 Gigs of data on my C:\ Drive and it needs 32 seconds. The total number of items is 1,300,000 on that drive, but it cannot complete. However, when I go down one level, as you suggested, it does complete. So, good job, Olaf. Thank you kindly for the help.


Dennis



Simplicity is the extreme degree of sophistication.
Leonardo da Vinci