manually reducing a dbf- and cdx-file

[Suburb]

Hello everybody,

we've an old Visual Fox Pro 6 program, that becomes very slow. We've no FoxPro programmer and the Company that programmed it bakrupted years ago.

We get the code from the Company and we installed Visual Fox Pro 9, but we can't compile the program with this Version (A module called "cryptor" seems to be the problem). The Producer told us, thet the size of the database should be the Problem. Normally there is a button to delete old entries in the database, but this button is disabled for us (it was our wish to disable a such dangerous button as the Producer adapted the program for us)

The database-files are encrypted, but we get a tool to decrypt and encrypt them. It's working very good. After decrypting we've load the database with the dbf-manager. But the Databsefiles are too big to delete entries in the databse with the dbf-manager. Everytime we try to delete entries, the Manager shut down.
We've tried to fix them manually in Editor. But after saving, we couldn't Import them in dbf-manager again. The Manager told us, that the files are encrypted.

Question: Does anybody know a good program (free or for real low budget) to modify the dbf- and cdx-files, so we can try to reduce them and hope, that the mainprogram accept them. Or does anybody has experience with reducing such files manually.

Thanks in advance
Suburb

 

[Olaf Doschke]

Cryptor is a well known product for encrypting (not only) DBF files and as that seems to have been used also is the only way out. From what you tell you have the end user version of it - must have it - to both encrypt and decrypt dbfs. How large are the files? The file size limitation is 2GB, if you hit that working on such files even via cryptor will hit that wall.

You might need a newer version of Cryptor to be able to use it in VFP9, too, or you need another programmiong tool also able to integrate whatever Cryptor version is included in your VFP application. That may be the core problem, as older versions of Cryptors DLLs are surely compatible to your files bot not compatible to VFP9.

Cryptor 5 -

http://www.xitech-europe.co.uk/cryptor5.php

- is said to work from 5-8, maybe also 9, but I remember there was some quirks with it. You may have Cryptor 4 or older, which existed at times of VFP6 already, that may mean an update to Cryptor 5 or you need an older VFP to work with the older Cryptor version or other programming tools. It does not only work in conjunction with VFP and the good news is the downward compatibility to the encrypted files - anything else wouldn't make sense anyway.

Maybe take a look at

http://fox.wikis.com/Topic/wc.dll?Wiki~Cryptor~WIN_COM_API

http://fox.wikis.com/Topic/wc.dll?Wiki~CryptorOnWindowsVista~WIN_COM_API

Edit: If only Vista (or newer OS) is the problem, you might fix the DBFs and transfer data to ordinary unencrypted DBFs within a virtual machine with XP.
If the OS problems are not the main problem, but a too old Cryptor version, you'd need to try from C++ or anything mentioned or get a Cryptor 5 license. That's a thing to discuss with Xitech, if their site is not only a non removed archive only and they still sell or assist.

What I know in detail about Crytpor is it hooks into any write and read of a file, so it is buried quite deep in the file system, you have a DBF file not only with encrypted data in char fields, the whole file included header is encrypted and no DBF reader tool can read it even just to read out the encrypted data for decryption with other tools. Since Cryptor also is closed source you may not even be able to decrypt DBFs knowing the encryption algorithm and keys, as it's an algorithm doing the en/decryption not in a way en/decrypting the whole filestream sequentially but at each blockwise read/write access. The Cryptor mechanism allows you to register files for en/decryption and that way of working with them works on any files, not only dbfs. And as said the Xitech site clearly states Cryptor also is integratable in other languages, too.

Bye, Olaf.

 

[GriffMG]

If you want a very simple, i.e. basic, program to enable you to pack and reindex your tables, I have one here:

http://www.finedata.com/support/browser.zip

You will probably need to download and run the VFP support files too

http://www.finedata.com/support/VFPRefresh.zip

To pack a table, once unencrypted, run the program from the same folder as the data,
the program will offer you a list of tables, click on the 'Exclusive' tick box, then
double click the table of interest.

You will be offered a list of index tags for that table, don't select any, hit the
exit button (bottom right)

In the next screen, type PACK into the bottom text box and click on the GO button to
the right of the text box and bingo, it should be compressed.



Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Olaf Doschke]

Well, no way to get a file decrypted in one VFP session and then act on it with an EXE being another process
A browser.EXE won't work and also isn't needed as VFP9 exists.

in case you can open a DBF unencrypted with the Cryptor related code along these lines, you can work on it further:

* Load the Cryptor 5 object
o= CREATEOBJECT("XitechCryptor.Cryptor")
* Use Deep mode to catch all modules in use
o.Deep=.T.
* Watch Visual FoxPro
? o.WatchDLL("VFP9.EXE") && Visual FoxPro 9.0 IDE
? o.WatchDLL("VFP9R.DLL") && Visual FoxPro 9.0 Runtime
? o.WatchDLL("VFP9T.DLL") && Visual FoxPro 9.0 Runtime MT

* If that worked so far
* Register a file with Cryptor 5 with a key and the en/decryption method (nMEthod=0,1,2,...)
o.Register("c:\path\to\some.dbf","key",nMethod)
USE "c:\path\to\some.dbf"
Browse

Now through the monitoring setup and registration of the DBF you should see the DBF in a BROWSE window unencrypted. You need to know the "key" password and nMethod number used to en/decrypt the DBF file from the applications old source code, perhaps.
Once you are at that stage you can COPY TO a new dbf or PACK or do whatever to the DBFs, but I'd certainly do backups first and rather copy out data to a new file than PACK the current one.

Bye, Olaf.

 

[Olaf Doschke]

If it's Cryptor 4 or ealier this may be helpful:

http://doughennig.blogspot.com/2008/04/using-cryptor-dll-functions.html

Because Cryptor wasn't an OLE class you can use the way I showed before.

Bye, Olaf.

 

[GriffMG]

Sorry Olaf, from what the OP was saying I thought he had the ability to unencrypt his tables to 'raw data' that could be manipulated
and then re imported

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Olaf Doschke]

Well, Cryptor doesn't work this way. If that would be the case, there wouldn't be a problem, would there?

Bye, Olaf.

 

[GriffMG]

The database-files are encrypted, but we get a tool to decrypt and encrypt them. It's working very good. After decrypting we've load the database with the dbf-manager. But the Databsefiles are too big to delete entries in the databse with the dbf-manager. Everytime we try to delete entries, the Manager shut down.
We've tried to fix them manually in Editor. But after saving, we couldn't Import them in dbf-manager again. The Manager told us, that the files are encrypted.

I think they have a tool that decrypts, and they have tried to edit the dbf files with something and have made a mess of it, leading to their program being unable to reimport them

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Olaf Doschke]

(A module called "cryptor" seems to be the problem).

Well, let's wait. I rather think they have problems either due to OS, hitting the 2GB size or anything else about it also Cryptor has a problem with.

I also assume the tool for en/decryption of data is built around Cryptor as your Browse.EXE is built around the BROWSE command.

An encryption only working on whole files is a bad protection, as that would mean files are decrypted at application start and put somewhere to later be encrytped. That's only protecting data while not using an application. Any worthwile encryption tool or feature works on the fly during use and the file itself stays encrypted all time. Same is true with a truecrypt/veracrypt container file, you can work on files in a veracrypt container and still the container file itself will remain garbage to mere mortal eyes only the veracrypt application itself can access via a given key and/or decryption key file.

Using Cryptor you can work on a registered file as if it wasn't encrypted at the same time opening it with eg low level file functions fopen/fread in another foxpro session or with notepad or whatever else you don't see the decrypted data. It's hooked inside the file system. The monitoring calls are registering VFP EXE and DLLs to route all their file syytem access through Cryptor code, which handles every packet of bytes read from or written to the registered files. Any unregistered file remains unchanged from that hook processing.

One can write such file system hooks officially with an WDK for minifilters. Software like antivirus works that way, too, and can hook into any file access to protect files. In this case the file blocks are modified and not just scanned. Since Cryptor does not act for any access but only from the monitired EXE or DLLs and registered files, it can restrict who sees the decrypted data.

Bye, Olaf.

 

[GriffMG]

Hi Olaf

I thought their reference to Cryptor was by way of how they would 'like' to move forward, taking the code provided
by the original programmer and recompiling it in VFP9 - not realising perhaps that VFP9 could work on the tables if they
were decrypted first - but not having the 'design time' version of Cryptor meant they were stymied.

I would suspect you are right and they probably do have a file size limitation, perhaps encrypted the files are smaller.
There is generally a lot of white space in .dbf files, so they do zip up rather effectively

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Olaf Doschke]

It could turn out the module not compiling just was called cryptor by pure chance. But if you have a license of cryptor the part doing en/decryptions is redistributable, perhaps just not the module you need for development and compilation.

I don't own Cryptor, I can't say, but most probably some copy protection shrinks down the end users to only work with such files, also create new ones, all under the end user license without the option to develop with this. Reselling software is possible under many jurisdictions, so the bankrupt old vendor could have sold Cryptor, too, then it might just be an issue of installing it for development. You still should be able to use the code also a developer has to use to create and work with cryptor files, otherwise there would be no way an end user could have such files and work on them. So the code for working on cryptor DBFs will work also with the runtime only variants of Cryptor, but you'd either continue without data encryption or use a new license of it or another tool, like VeraCrypt, once you copied out the data from a DBF registered to a new file not registered with Cryptor.

The situation with the Cryptor DLLs/FLLs is the same as with VFP Runtimes, you can run VFP code with the VFP Runtime, but you can't write new VFP code and build an EXE without the VFP9.exe code IDE, which isn't redistributable (technically of course, but not legally). And so they might only have the Cryptor runtime and not the full product for use in building an EXE.

Bye, Olaf.

 

[Olaf Doschke]

One last thought: If the DBFs have a corruption like any normal DBF could have, too, the way Cryptor can be wired to some EXE or DLL may also be used to wire it with FoxFix or some other DBF repair tool to work on the corrupted DBF. The hope is, the corruption also 1:1 is correctly encrypted and decrytable. If that's not doable you'd have a hard time with a defect encrypted DBF.

Bye, Olaf.

 

[GriffMG]

It seems, if Cryptor is the software being used, that the company XITech has ceased to exist (their web site remains)
and the software may have transferred to SAP

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Suburb]

Thank you for all the nice Feedback

The mainproblem is not the en/decrypting the files (we've a Special tool with cryptor included, that decrypt the dbf- and cdx-files). Our Problem is the manipulating of the human-readable files.

But the hints with the newer cryptor-version might solve our Problem of compiling the sourcecode. We will Keep an eye on it.

The files are by far not 2 gb. The biggest is ~200mb. The most files are just around 20mb. There should not be a problem with the 2 gb Limit. Nevertheless, the whole program is working very slow, cause of the many datas it is connecting. The program/sourcecode is from the late 90s (looking at the comments). I think, it is miles away from a modern, fast programming style. The lack of Speed with a bigger database is a known topic of the program.

We a searching for a quick and dirty (and cheap ) solution to use the program a few month more. I think sooner or later we will program our own product.

Thank you for your Attention and the suggestions. First of all I will try the packing and indexing program.

I will report if there was something, that solved my problem (Otherwise, we will start to program our own product. It's not my decision)

P.S.: The OS is Microsoft Windows Server 2008 R2 Enterprise

 

[GriffMG]

If you have bought and installed VFP9 then that can do the pack and reindex on your decrypted file.



Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[GriffMG]

The OS is Microsoft Windows Server 2008 R2 Enterprise

If it's slow, that could be op' locking on the server.

Turn that off on the server and the workstations - it could make a difference

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Olaf Doschke]

The most probable cause is oplocks, if the problem started with 2008 R2. That's true.

After decrypting we've load the database with the dbf-manager. But the Databsefiles are too big to delete entries in the databse with the dbf-manager.

That sounded like the decrypting had an influence on the size and the size was the problem of deletions.

If that tool writes out decrypted DBFs it misses the point of Cryptor to work on the encrypted data, while only the application EXE and VFP runtime see it decrypted.
But if that is the case you only need the PACK command on a DBF decrypted to PACK it, REINDEX to reindex it. I don't know what dbf-manager you talk about, as VFP developer you have VFP itself to manage DBFs. You are free to do anything with a DBF via VFP. You can do PACK and REINDEX (non destructive removal of garbage, PACK includes REINDEXING, by the way, as the end nodes of indexes need to point to the renewed recnos of the packed DBF), and DELETE (well, deleting records),...

But as deletion was no feature of the application PACK won't resize, as there is nothing to remove and downsize the DBFs, so the first step would be to delete something. You have to understand the database model and relations between tables to delete everything belonging to some work item, order, employee or whatever entity you want to remove, though. You can delete within a browse window, you can delete via DELETE FROM some.dbf WHERE condition, that's the SQL way - or you DELETE within the open DBF via DELETE NEXT 1, DELETE FOR condition, DELETE WHILE, whatever.

In regard of continuing development, you might cut out all things related to Cryptor, if you don't have inherited that license and can't compile, so you don't need it. That'll just mean data will be readable. You can still limit access via Windows File system privileges - the normal file/folder security. Access to a DBF via decryption slows down things by the en/decryption, obviously, too. So removing that feature can enhance performace, I'd not expect much of that, though, the main profit will be to be able to compile again and continue. If what Griff said about Xitech is correct you only have a chance contacting the bankrupt company again for their Cryptor license to be able to compile again. HalloGram once was a company selling also Xitech products. At least the last sign of activity there is their copyright 2014 notice and since their core products now evolve around barcodes, their involvement around Foxpro might be over, but they still exist.

Bye, Olaf.

 

[GriffMG]

It's interesting, the company was based just a few miles from here, I did think of knocking on their door.

But then, I thought better of it.

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​