find non vfp file open in a non vfp application

[AlliMac]

In thread 184-1674267 the poster asked how to tell if a file was open. After a bit of prompting he/she explained that it was an Excel file that was "open, meaning visible in your taskbar".

Mike Lewis offered a solution using APIs, GriffMG offered a solution using low level file calls and there was other advice.

I don't know what I'm doing wrong but the low level file calls don't work for me as hopefully is clear from the following:

With the VFP IDE open, use file explorer (or whatever) to open, say "C:\MyFolder\MyFile.txt", a file with some text in it.
Return to the VFP IDE and enter the following (and get the following results)
public gnHandle
public gnBytes
m.gnHandle = FOPEN("C:\MyFolder\MyFile.txt", 12)
?m.gnHandle && 21 (For example)
m.gnBytes = FWRITE(m.gnHandle, "This is the house")
?m.gnBytes && 17
?FCLOSE(m.gnHandle) && .T.

Switch to the open file "C:\MyFolder\MyFile.txt" - there is no change.
Either: close the file without editing it, open it again and there is the "This is the house"
or: edit the file, close and save and then reopen it and there is no "This is the house"

I don't understand why what I get seems to be so much at variance with the advice given, especially from Olaf Doschke, a man with whom I would be very afraid to start an argument. From what I read, I expected the FOPEN() funtion to return -1 if the file "C:\MyFolder\MyFile.txt" was open in another application

Mike Lewis's API solution worked for me, but I also need to know the path name for the file. My knowledge of API usage is very limited, I use them strictly on a "monkey see, monkey do" basis.
However I found 3 possibilities, GetFinalPathNameByHandle, GetFinalPathNameByHandleA and GetFinalPathNameByHandleW. No idea what the difference is. I tried them inside Mike's code. My declaration for all of them was
DECLARE integer GetFinalPathNameByHandle IN WIN32API ;
integer hFile, string lpszFilePath, integer cchFilePath, ;
integer dwFlags
Same for the ...A and the ...W

My usage was
lcBuffer = REPLICATE(CHR(0), 250)
GetFinalPathNameByHandle(hNext, lcBuffer, LEN(lcBuffer), 0x8)
Same for the ...A or ...W. hNext is the handle returned for the file in Mike's code.
This failed. I managed to paste in GetLastError() code from an ancient VFP knowledgebase article. This told me the error was 6 "The handle is invalid"

Could someone please tell me (1) what I'm doing wrong with the low level functions, (2) what I'm doing wrong with the API functions and (3) How to find out if a non VFP file is open in a non VFP application, and if so its path.

If I haven't yet bored you all to death, the practicalities of my question are as follows: My (tiny) application frequently writes to a file named user.log as a means of maintaining an audit trail. The user can view and edit the log from a shellexecute in the application menu. I want to cover the situation where the user (or for that matter anybody else with access to the folder) may have opened the log and perhaps forgotten about it. I want to be sure it is closed in any other application before it is written to by mine. If it is open, perhaps being edited because it has become to big, then tough, out it goes. If there is another application with a user.log and it is open, I want to leave that one alone.

Thanks for your patience

 

[mjcmkrsr]

Hi,

You may want to FFLUSH(m.gnHandle) before FCLOSE()

hth

MarK

 

[Mike Lewis]

Here is the link to the original thread that AliMac is referring to:

thread184-1674267

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Mike Lewis]

Mike Lewis's API solution worked for me, but I also need to know the path name for the file

Unfortunately, my solution can't help with that. My code doesn't actually look for a file. It looks for a window which includes a specified string within its caption. That string could be a filename, but that is not an intrinsic part of the solution.

Griff's solution might be a better possibility.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Chris Miller]

We had a recent thread here...thread184-1820122

But before you go there, what are the consequences of your experiment with a txt file? You see that notepad does not keep a file open, it reads it in and works with the in-memory copy of it.
Thus changing the file from another process is possible, does not influence the text in notepad, and if you save from notepad, it just overwrites the file, because it also doesn't care that the file was altered in the meantime.

In short: Getting a file handle isn't a mechanism ensuring no application is currently having the file in memory and in its windows controls like a canvas or edit box or whatever for editing of users. It's just ensuring the file is currently not open with a file handle, the low-level key to the file itself.

So it's not a two-way solution. It's that way: If you get no file handle for write mode you are sure something else has a handle in any mode, read or write or both. But if you do get a file handle it says nothing about an editing state. Since an application most of the time works with the loaded file content in memory, usually, even with a representation of an object tree with references, etc. in memory, there is no lock nor a handle on the file that would help you to see the file is worked on.

So in the end you can only be sure about situations where another process keeps the hands on a file and does therefore not let you open it in write or read/write mode. You could usually get a handle for read mode only, but you'll read the file, not the notepad text editor or Excel sheet or any other applications' visual representation in its current state. Only if applications sync the file state with what they display all the time, you would also read that from the file.

In the end, there rarely is anything that works on the hard drive or filesystem-level during the modification you do within an application. An application internally works with the file loaded into memory and some applications may keep a file handle open to ensure no other process can change the file they let its users work on. Other applications, like Notepad, don't care about that. Notepad++ is a bit different from Notepad in that it allows changes from outside, but if it detects a change in a file you currently edit (it does so with a time lag) it will ask you whether you want to reread that file and see its current content.

So, all in all, it depends on how applications are programmed to work with files. In VFP you can also choose how you work on DBFs, with or without manual locks, for example, buffered or not. But even in a non-buffered state, you don't change an FPT memo content with every change of an edit box. The moment the memo is modified in an FPT file is when valid is triggered by setting focus to the next control and the valid event or base behavior of valid returns .t.

A situation where the file is already exactly what you see within an application is as rare as it gets, I assume some hex editors could easily work that way. It's not necessary for the file handle mechanism to work. For that, it would be sufficient if every application would just keep a handle open even if the application doesn't keep the file in the current state displayed in itself, but just to allow such a low-level mechanism to detect this file is in use. But not all applications do that. In VFP you can also use FILETOSTR() and have a copy of a file in a string variable, but there is no file handle kept open you need to close and there is nothing forcing you to finally also do a corresponding STRTOFILE to write changes back.

There is no pairing like that necessary. And that's independent of programming language or file system. It's just a decision of how it's programmed for an application. It could also use a mechanism that's only known to itself and that only other instances of the same application would detect. There is no general low-level OS mechanism that tells you whether a file is "in use" in the sense some application will overwrite it with "save" from its file menu or similar options the user has to decide when to persist modifications in the original file. Editors easily let you save as some new file, which alone is a reason not to act as the current "owner" of a file currently edited.

Chriss

 

[GriffMG]

My suggestion relies on the target file actually being open. With Excel and Word this is the case when you open a file
the application leaves it open while you work. If the file is opened in notepad though - it is read entirely and the open
file released. At this point a copy of the file is held in notepad's memory, so my method will not then work...
well, not as expected anyway

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Mike Lewis]

Just to be sure exactly what the problem is ...

Are you saying that your application can read and write the log file; that users in other applications can also read and write the file; and you want to be sure that there are no collisions (one set of edits overwriting another) and also that neither application will be prevented from opening the file.

If that's right, I would suggest that a text file is not a good choice. As the others have indicated, there is no way for an arbitrary application to lock a text file.

As an alternative, consider replacing the text file with a DBF. Create a simple, stand-alone log viewer program that does nothing more than let the user view and edit the DBF. Both the log viewer and your main application would then be able to apply all the familiar record locking and exclusive use techniques to prevent collisions and to control lockouts.

Of course, you would have to stipulate to your users that they would always have to use the log viewer whenever they want to access the log. But they might see that as an advantage, given that the viewer could give them more user-friendly features than they would get from working with the raw text.

Just a thought.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Chris Miller]

Ah, yes, I forgot to recommend something for the case you want to cover.

Well, the only way I see to let your own application know a log file is in use is that you don't just use a shellexecute and let notepad display that text file, but let that be a viewer application by yourself. Then you can open it "exclusive" by using fopen in mode 12. And then no other application will be able to use the log file until that viewer closes it again.

So you can stick to a txt file, you just have to have your hands on all access to it in a way that a non-wanted change is communicated more or less direct. In case your viewer has the log open in mode 12 there is no chance for any other process to get file access. But you can't know if some user uses notepad to look at the log file and you could only prevent any writing over of new log entries you added while someone edited it in notepad and finally overwrites it, if you keep hands on the file all the time. You usually don't.

The problem of a log file getting too long is usually to only let it grow to a maximum size and then archive that and start a new log file. That way you would remove the need to open the file to empty it. Old archived logs then can be deleted, or whatever suits you. Backed up, zipped, whatever. That's how I would tackle that part of the problem.

What would suit you, I guess is a pause mode, every logging stops or is redirected while a log file is inspected. And after the inspection including perhaps some edits finishes, what would have been written to the logs initial state will then be appended to it.

With your own viewer you could of course do that: Rename user.log to paused.log for editing. Create an empty user.log the main application then uses, let users edit paused.log and at the end of the editing save to paused.log and append the current user.log to the edited content and you have those new log entries "merged" with the log edit. In this case the merge is just an append, as all entries in the newly started empty user.log are after anything in the user.log you renamed to paused.log.

In short, that's what a snapshot mechanism does, at least one way of doing such a thing. You can't instantly copy the user.log to a paused.log, but you can risc the short interval a user.log is renamed to paused.log and no new empty user.log file exists. Even more so, if you program your logging in a way that it adds to a user.log and due to a missing file it creates a new one.

The joining of what was added to the user.log after the editing is a step you surely can't automate if you keep using ShellExecute or notepad for the log editing, that's a step you need to program into the save mechanism of your own log viewer/editor.

Chriss

 

[Chris Miller]

There are tons of other ideas, I'm sure you can come up with one of them yourself. Don't lock into a problem that's not the actual issue you want to solve, just think about other means. Another simple one would be you could put your application in a mode it would log into memory instead of a file and when you switch back to normal mode it would add what it logged into memory to the file log. That way you simmply avoid the need for the user.log to be available all the time for the application logging.

Chriss

 

[Chris Miller]

And last not least don't underestimate the simplicity of Mike Lewis recommendation to base the log file on a dbf. Yes, that makes viewing harder, you now actually need a viewer that shows the log, in the simplest case the dbf has a 254 char field and you show it with a browse or a grid. Editing paragraphs, selecting rows for deletion etc. then is a challange to provide, but you can easily look at a dbf while the main logging feature adds new records to it.

Chriss

 

[GriffMG]

With a dbf log, if you want to cut and paste, simply write it out to a text file and read that.



Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

Griff, of course selection copy & paste is easier in a text editor, you can also concatenate the log records to a single string and bind an editbox to it instead of displaying the rows in a grid or browse no need to go to a text file first.

Anyway, a "cut" operation is not simple in a dbf, because of the nature of deleted rows just having a deletion mark.

I also don't know what size we're talking off, a log that has a GB size is not easy to handle in any way

But as far as I understand the need to cut rows mainly is to cut off the oldest and first log entries to make room, and that problem is clearly much easier to tackle with a segmentation like only allowing a log to grow to 10MB and then create a new user1.log,user2.log, etc., perhaps even cyclic so that in case you reach user9.log your next log is user1.log deleting that file. Depends, how old log entries you really need. You could also write seperate logs for messages, warnings, errors and flag records with a severity level or any other means to categorize them.

Chriss

 

[Mike Lewis]

I've just remembered that I actually wrote a log viewer in VFP about 10 years ago. The application recorded all its inserts, updates and deletes into a log file, which was a DBF. In the viewer program, the user could drill down by user, date and transaction type. The results were then shown in a grid. The user could select which grid columns to show, change the order of the columns, and make similar adjustments.

I think you will agree that this gives the user a lot more functionality than simply viewing the log as a text file.

But it would have been easy to also provide an export feature, to allow the user to copy some or all of the data to a text file, or a workbook, or whatever.

The 2GB limit was an issue. To solve that, the application periodically checked the size of the DBF. As it approached, say, 1.8 GB, it would delete about 0.5 GB of the oldest records, and then pack the file. This was done during the application's routine monthly maintenance, which permitted exclusive use.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Chris Miller]

I don't know how urgent or not the problem is for you, but in summary: Your idea of detecting whether a file is open in any application clearly fails.

Because, in short: A file can be loaded into some application without the file being open with a handle on it, not even a read handle.

So solve your overall problem by logging differently in the first place or providing the full lifecycle management of logging by not only writing the logs, but also offering a log viewer and editor. It's quite straight forward to rather rethink the solution from the bird perspective than being stuck in a detail you can't figure out to solve, just because you think it's just the last little detail you need to solve.

Ask yourself what you would do, if you could detect a file is open in the sense you want to know it for: So you know the application could overwrite it as the save step of its editing. What are your means against this save step? Unless you keep the log file open exclusively and thus already prevent the situation to come up, you can't prevent your new log entries are overwritten by the editing of the log. So even if you solve your detection problem, you're still far away from a solution. aren't you?

Contradict me and tell me what you'd do, if you knew notepad has the user.log file open. What then?
Edit: Notice even if you get no write handle on a file and know it is open right now according to the filesystem, you still don't know which process has the handle that hinders you to get write access to the file.
End of edit.

Do you then know you need to wait for further logging until that editing ends? And what if the user editing leaves the log open and forgets about that? Even until shutting down the workstation? You have to log somewhere else to prevent all current log entries never being persisted at all, don't you?

I do understand you explicitly want to enable editing, so just preventing that is no solution, but you have to have your hands on the full lifecycle of the log, so a log viewer is a natural answer.

There lurk other problems: Files could always be deleted or moved for any reason, by antivirus software, for example, into quarantine. So if a log is more important to you, you'd rather go for using data files for logging or even Windows event log. By the way, Windows offers an application called... Event Viewer (*tada*). Why do you think it does? And why is the Windows Event Log not simply a txt file with all its vulnerabilities?

Chriss

 

[Mike Lewis]

AliMac,

It's nearly a week since you posted your question. You've had around a dozen replies since then, with lots of advice.

It would be really helpful if you now gave us some feedback on those replies. Were any of them helpful? Were our assumptions correct? Was our advice on the right track?

If nothing else, it would be a courtesy to acknowledge the help you have been given. A simple "thank you" would be enough.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Chris Miller]

There is one last simple thing to answer we owe to your initial post. You said you get the error 6 from GetLastError, which you looked up translates to:

"The handle is invalid"

That's the correct translation.

You can know this in advance. If FOPEN or also if FCREATE returns a negative value, that's not a valid file handle. So before you assume you have opened (or created) a file, you have to check if the handle you get is positive. If you get a negative value back the FOPEN failed because something else has the file open and you then don't need to and can't FCLOSE it.

I can also tell you when you get this. We already said notepad reads in a file and then edits what is loaded in memory. If the log file is big enough, the reading phase takes a while. The more frequently you log, the more probable your logging overlaps with that reading in of notepad and you get no write handle during that time. STRTOFILE would also fail in that case. Both won't trigger an error event, though. FOPEN returns -1 and STRTOFILE would return 0 as the number of bytes written.

When you finally get a valid handle, that's not because notepad has finished editing, though. that's when your logging succeeds again, but will be overwritten, if the user saves his edited notepad text.

As I already said, it doesn't help you to get a handle to know if what you write to the log won't be overwritten.

does it all become clearer now? I can also draw you a timeline of events:


1. User opens user log manually or by your application via SHELLEXECUTE user.log. Anyway, it opens in notepad
|
| notepad load progresses
|
V
| 2. FOPEN: returns <0 value
| 3. FRWITE fails with a negative handle, you don't have a valid file handle, your app crashes or continues ignoring the error
|
| notepad load progresses
|
V
4. Notepad finished loading
|
|
V
|
| 5. FOPEN, FWRITE, FCLOSE succeeds, you log new lines
|
V
6. user edits log file in notepad
|
| 7. FOPEN, FWRITE, FCLOSE succeeds, you log new lines,
| 6.+7. can repeat many times.
|
V
8. user edits in notepad
|
|
V
9. user saves and overwrites your log entries
|
| notepad overwrites the file.
|
| 10. FOPEN gives handle<0, FWRITE fails. That can repeat many times, again.
|
|
V
11. notepad saved

After that logging continues normally.

You lost not just 2, but as many log entries as were made from the point notepad has loaded fully and the user saved. And you lose the log entries during the save period, too. So you will not only have the edited log file but lost log entries that would have been done during the editing phase.

The longer the user edits - and in the worst case he forgets that and does other things the whole afternoon, for example - all that time the log entries you make will be overwritten, when the user finally saves in notepad.

Maybe now you understand why both me and Mike suggest you solve that by providing a log viewer/editor yourself. So you know the state of the log all the time. See ideas in previous posts.

You can't say by having a valid file handle, that the file is now in your hands, fully. It is temporarily in your hands, but notepad can also have it loaded currently, and when the user saves, that overwrites all you write to it during his editing.

The reason a DBF works so nicely is that you can INSERT (or APPEND BLANK & REPLACE) new records while one or even multiple users look at the DBF records in a grid and edit it. All the necessary shared file handling is normal for a DBF file. Then you can provide a feature to delete all entries previous to some datetime. Or select records with a specific word in them, or whatever else users and you yourself need.

Last not least, even now you know how the FOPEN function works (it's all documented what the return value means not only in the normal case when it's the file handle, also when it's negative!), you still don't have a way of full control over the log, if you let users edit it with notepad.

When the log file is small, the time between point 1 and 4 is short, likewise the time between point 9 and 11. Then you still have the problem of all the time between 4 and 9 the user edits while you add log entries, that finally are lost.

Chriss

PS: I wrote a short prg that writes a large file with 1MB size spaces and then adds 20 lines per second with FOPEN,FSEEK,FWRITE,FCLOSE. FOPEN returned a good handle also when I opened the file in notepad, even though it took some time to load as it was initialized to 1MB size. But I got a failed FOPEN when I saved an edited file from notepad. I would have expected more fails of FOPEN, but VFP seems to wait, if it can't immediately get a handle,, that would explain the error tolerance. Nevertheless you get into situatons you don't get a valid file handle, more likely during saving of the edited log, I guess, as you need a write handle that's posessed by notepad in that case and is more problematic than one process only reading while the other wants to write.

 

[Chris Miller]

Last response to

If there is another application with a user.log and it is open, I want to leave that one alone.

The solution to that is have a separate user.log, unless you're having a central log for all clients. Well, that again is better to handle with a central DBF, that's VFPs thing to persist data, isn't it? With all it's ways to use tables shared and handle concurrency and conflicts, transactions and/or buffering and/or locking.

The big advantage of a txt or log file with just Ansi text is the editing with any editor, but the disadvantage more than destroys this advantage.

Chriss

 

[AlliMac]

Okay, okay I get the message. Using a text file as a log file openable in Notepad is a bad idea.

Thanks for your response mjcmkrsr. I tried fflush() but it doesn't help if another user has the file "open" in Notepad, edits it after the fflush() and then saves his/her edits.

Thanks for your response Mike Lewis. My application writes to the log file and the user can read and write the log file. But of course it's just a text file so anybody with access to the log file's folder could open it also, inadvertently, curiously or even with malicious intent. And yes, I now understand that your code gets a handle to a window not a file, as does some other code I have played with.

Thanks for your responses Chriss Miller. Clearly I didn't know how Notepad (and other applications) work with their "open" files. However it makes it pretty clear I can't rely on getting a file handle to tell me if a file is open in Notepad or not. I had a play after reading your first response and found that I could open a text file in Notepad a number (limitless?) of times by double clicking the file name in File Explorer. They could all be edited and what was actually saved was the editing of the last man standing.

I have been drafting this reply for several days now and keep revising it on getting more responses from Chriss Miller. I appreciate all the time you have put into this Chriss, thanks very much. I certainly have a better understanding of how Notepad works and I can see it (and probably a number of other non-VFP applications as well) is not a good way to log application activity. It all seems to be summed up in your line "Because, in short: A file can be loaded into some application without the file being open with a handle on it, not even a read handle".

Thanks for your response GriffMG. You and Chriss Miller have explained to me how text files and notepad work. I suppose I have never had the urge to open the same text file in Notepad multiple times or I might have worked it out for myself. Never mind, I know now thanks to your responses. I played with the code you posted in thread 184-1674267 and of course I found that although it doesn't work with text files open in Notepad, it works fine for things like spreadsheets and VFP tables. Your cut and paste suggestion needs thinking about too.

So now it's back to the drawing board to come up with a logging system (for errors alos) that is "inside" VFP. Another couple of tables might be on the horizon.

Thanks to all the responders, particularly Chriss Miller. I've saved the entire post to (dare I say it) a text file so I can easily read it over and over again. Thanks also to Tek Tips, a great forum.

 

[Mike Lewis]

Just another point to consider ...

Is there a good reason to allow users to write to a log file? I can understand that they must be able to read it. There would be no point in making a log if people couldn't see it. But if the aim of the file is to log events within the application, why would anyone be allowed to write to it?

This is not a criticism. You know your setup better than me. But it might be a point to keep in mind.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[AlliMac]

Thanks for your response Mike. The only reason a user would want to edit the user log would be to chop the top off it to bring it down to size. Whether or not this would actually ever be an issue in my tiny application is a moot point. I was really thinking of inadvertent editing or access.

This thread has given me a lot to think about. I thought I was being very clever using a text file and shellexecute for the user log (and the error log and a couple of others). No form or table required, very simple. However I can now see that the disadvantages outweigh the advantages.

So far my thoughts are a table maybe with one field since all the entries generated by the application at the moment are in date and time order anyway, maybe updated on the fly, maybe batch updated when the user logs off. If the latter does it matter if the application crashes and the batch update doesn't happen? Perhaps make an entry on logon so at least there is a record of the crash (no logoff). Probably add a bit of code to delete the first nnn records if the reccount() is more than nnnnn. Lots of ideas suggested by the thread and prompting me to come up with some ideas of my own.

As a final thought, my application has been using text files for logging errors and activity ever since I first created it twenty years ago. I can't remember when I implemented attempting to close a log file that might be open by another user, but I recently discovered that due to a coding error, it never worked anyway. This proves at least that my testing was faulty but it might also suggest that nobody has ever been interested in the log files anyway. However I will persevere if only to keep my brain going.

Thanks everybody.

Thanks everybody who responded.