.dbf files mysteriously gone missing (or being deleted)

[Vince Wen]

I have a VFP9 program where users can install on either their local workstation (C or put on a network server for multiple users to use.
There have been several instances (perhaps 3-4 instances this year) where .dbf files have mysteriously gone missing. It is a different .dbf file each time, and there does not seem to be a pattern.
As far as I can tell, the cases have all been happening when the software is on the network server. I have a case where the user was using the program just fine and suddenly it errors with: "File [xxx].dbf file does not exist". This doesn't make sense to me as the user had to be accessing the .dbf file to run the program, and I certainly don't have any code that would delete the .dbf file.
So far, this has only been happening to files on network servers. It has not happened to our C: installations. Of course, this could just be a coincidence.
Does anyone have any hypothesis for me? Or any advice as to where I should begin my investigation?
- Vince

 

[SoftArt]

Vince,

have you run a virus scanner and checked whether the dbf files have been quarantined?

Thomas

 

[Chriss Miller]

My hypothesis would be Antivirus/Antimalware (perhaps running server side), putting files into quarantine and thus removing them from the file share.

It's essential with any file based db system to make data files an exception of such malware protection. Not only does it harm in terms of concurrent access, causing write operations to fail, in the worst case dbf and or fpt (memo) and cdx (indexes) corruptions. But also a rule to remove a file to quarantine is desastrous.

Ideally you'd not just exclude directories but a whitelist of known filenames, but that comes down to practicability of defining the virus scanning.

 

[EinTerraner]

Do You mount the folder that is shared on the server as a local drive using "NET USE".
like "NET USE X: \\ServerName\SharedName"

 

[Vince Wen]

Do You mount the folder that is shared on the server as a local drive using "NET USE".
like "NET USE X: \\ServerName\SharedName"

No, we do not.

Vince,

have you run a virus scanner and checked whether the dbf files have been quarantined?

Thomas

Hi Thomas,

We run our installer through virus scanner (Virustotal?) prior to distribution. We pass.

Sounds like you and Chriss both have similar hypothesis. I will try to check with the users' tech support to see if any of our files were quarantined on their server. Unfortunately, I cannot check their server side as I don't have access/privilege to their servers.

 

[Vince Wen]

My hypothesis would be Antivirus/Antimalware (perhaps running server side), putting files into quarantine and thus removing them from the file share.

It's essential with any file based db system to make data files an exception of such malware protection. Not only does it harm in terms of concurrent access, causing write operations to fail, in the worst case dbf and or fpt (memo) and cdx (indexes) corruptions. But also a rule to remove a file to quarantine is desastrous.

Ideally you'd not just exclude directories but a whitelist of known filenames, but that comes down to practicability of defining the virus scanning.

Hi Chriss, I will try to check with the users' tech support to see if any of our files were quarantined on their server. Unfortunately, I cannot check their server side as I don't have access/privilege to their servers.

 

[EinTerraner]

How You access those files?

 

[Chriss Miller]

Business suites of Cybersecurity may or may not run serverside or local on all clients or both. Where files are quarantined thereofore is also depending on how their suite works. One thing is sure: Whoever is responsible for their cyber security should know more, get warning messages and have an overview over quarantined files.

The only VFP commands that actually delete a DBF and create a new one are PACK and ALTER TABLE, PACK creates a new DBF file, moves all undeleted records into it , then deletes the old dbf and renames the new. If that fails on the last step you'd see some new dbf file, though, maybe not yet with dbf extension. ALTER TABLE likewise creates a new DBF with altered structure and moves data over to finally delete an old DBF and rename the new one. If you assume PACK or ALTER TABLE actually modify the existing DBF files you assume wrong. But both mechanisms would need to fail in the last step of renaming the new dbf after the old one was deleted (and corresponding fpt,cdx files), so in both cases you'd find other new files. VFP would not delete the old dbfs before new files were created and filled. Also, those are operations requiring exclusive access, so it is quite unlikely you use these commands in your software at all. If you do, you might reconsider your approach of handling your data.

Edit: Of course there are more commands to delete files, like ERASE and DROP TABLE and maybe even more I overlooked, but you'd not be surprised about missing DBF files if you use those commands explicitly.

Anyway, all that makes the quarantine hypothesis much more likely.

 

[A_Radiolog_inFoxPro]

Someone is maliciously deleting them.
Check your logs.

OR

The program has some subroutine that deletes / replaces said file and it errors out half way after deleting and before creating it again.

 

[Chriss Miller]

Someone is maliciously deleting them.

Surely possible. I guess EinTerraners questions are trying to understand, if users actually see a mapped drive and know where files can be found and then be manipulated. The question then is what's in for a user that wants to use the software to delete a dbf?

 

[A_Radiolog_inFoxPro]

if users actually see a mapped drive and know where files can be found

Since the software is an a remote share. By definition the files including prg and dbfs are visible to the user.
Either if the whole directory is mapped as a shared drive

Or the shortcut on the users desktop is pointing to the shared path that is not visible to the user as easy.

Its still able to be found. ( if the user has some level of windows knowledge.)

Unfortunately there is no way around this. For foxpro. I haven't found a way to keep these stuff separate from the USER.

 

[EinTerraner]

You also can access all filees also by the whole Networkpath like

USE "\\FarComuterName\SharedPath\Userpath\Filename"....

DO "\\FarComuterName\SharedPath\AnyProgram" with "\\FarComuterName\SharedPath\Userpath\Filename"

therefore my question how You do the access on those files.

 

[Chriss Miller]

A software can use UNC paths and those will not be seen by users, necessarily. So you create a share, name it, but don'T map it, then your software can use files without the user knowing about them.

Its still able to be found. ( if the user has some level of windows knowledge.)

You're right about that, but that would point out a user would need to have more knowledge about how to find out which files a software accesses. If you'd do it that way, I myself, even though I know this mechanism exist, would perhaps get to grips to list available shares on a network, also those not mapped to drive letters, but the normal Windows user opening up Windows Explorer wouldn't see them.

 

[EinTerraner]

A software can use UNC paths and those will not be seen by users,

Set this UNC-Path as an environment-parameter like "FarDataServer = \\FarComputer\SharedPath" and you can access them with

Lc_FarData = GETENV("FarDataServer")

To access them by explorer just enter the path "%FarDataServer%"

 

[EinTerraner]

where .dbf files have mysteriously gone missing

One Question. Does this missing tables have memo-fields (.fpt) and/or indexes (.cdx) an are those also missing?

 

[Chriss Miller]

EinTerreaner - you have to know this. You'd not be stupid to set your share name as environment variable.

You'd need to be able to run process monitor and monitor which files a process accesses. You'd need to be able to write code listing shares. It's not a safe/secure way to protect files to not map them, but the normal user won't care where files are, even the EXE itself, as long as a software works.

Look, I'm not saying that users don't know the location of files, Vince Wen will know that, maybe his software even asks for a location to store the database and user then know by definition. Anyway, the question what's the benefit for a user to delete its data remains unanswered. The mechanism that explains missing files without suggesting malicious acting of somebody is still quarantining. Modern malware working on heuristics will suspect things that's normal for database software: Repeated random access of files. They don't need to contain a virus signature to make them suspicious and get quarantined. We'll perhaps hear back from Vince, whether that's the case.

What you could do against deleting files is removing that permission, you don't need delete permission on a dbf and all related files to work with them. It would also have the benefit that if your software somehow indirectly does try to delete a file it would encounter errors and you'd log them with error handling and find out.

I wouldn't know where to look in any system logs to see which user deleted a file, I don't think that's protocolled into the Windows Events log, for example.

 

[ggreen61]

In order to prevent a user from being able to browse to a share that contains the tables, I have used the following strategy:

//sharename/protectedfolder/sharedfolder

The protected folder user rights are not granted to the business users. The sharedfolder user rights is granted to the business users. The full path to the sharedfolder is hidden in an ini file that has the value encrypted to prevent viewing. This prevents a user from being able to input the full path to the sharedfolder. I do not map a drive to the shared folder; I always use the share name with the path.

 

[EinTerraner]

You'd not be stupid to set your share name as environment variable.

The sharedname was only mentioned as an example. In my systems i only have the "servername" for local network or the "IpAddr" for far networks in the environment. My DataPaths are variables inside my application.

 

[A_Radiolog_inFoxPro]

In order to prevent a user from being able to browse to a share that contains the tables, I have used the following strategy:

//sharename/protectedfolder/sharedfolder

The protected folder user rights are not granted to the business users. The sharedfolder user rights is granted to the business users. The full path to the sharedfolder is hidden in an ini file that has the value encrypted to prevent viewing. This prevents a user from being able to input the full path to the sharedfolder. I do not map a drive to the shared folder; I always use the share name with the path.

YEs but since you need windows to access the shared path directly to make it avalabel to your aplication

means the local user uses the creeds you have in your app to connect to said share.

So if he can find out from computer management the open files he will be able to browse to said file. So you are back to square one.

It's literally a chicken and egg problem.

the only solution I found was to have a windows server running terminal services and launch the application on full screen without the ability to minimise or manipulate the desktop environment.

But If your user just connects with a proper session on your terminal services desktop environment and opened the RDP shortcut with notepad he can see the full path of your app and fully manipulate it.

 

[Chriss Miller]

So if he can find out from computer management the open files he will be able to browse to said file. So you are back to square one.

That's basically not the problem in a company environment, where users have limited permissions. Using computer management needs admin permissions.

You still have a point. The simple solution to avoid deleting files is not giving that permission to users, which obviously still depends on them being non admin users. In a comapny that's very usual, though.

If you're concerned about the safety of your data the short answer is not to use DBFs but a MSSQL or MySQL or PostgreSQL backend or whatever else.