We recently installed a velociraptor hard drive on a quad-core Dell computer to speed up the processing time of our VFP programs. We have programs that run 16 hrs and more. The bench tests for the raptor drives were phenomenal particulary the transfer rates. However, when we executed one of our VFP programs there was no difference in the amount of time it took to process. In fact, in some cases the VFP program ran a little slower then it did with the original hard drives installed. Does anyone know if this is a VFP issue? We use version 9.0. Is there something we have missed in regards to configuraton of VFP or the harddrive? Thanks in advance.
Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations derfloh on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
VFP Performance on Fast VelociRaptor Drives
- Thread starter CVN75
- Start date
There are many factors involved in VFP processing speeds. Usually, optimizing of the data or more particularly, the indexes themselves is the biggest factor along with whether you have exclusive access to the files at processing time or not.
Another issue that could come into play is where your original data resides - i.e., where your temporary/intermediate files are written.
And on top of that, if there is any sort of network involved, throughput and/or hardware can be a major factor.
Without code samples of what you're processing or knowing where your files are located though, it's hard to guess what could be improved.
-Dave Summers-
![[cheers]](/data/assets/smilies/cheers.gif "[cheers] [cheers]")
Even more Fox stuff at:
Another issue that could come into play is where your original data resides - i.e., where your temporary/intermediate files are written.
And on top of that, if there is any sort of network involved, throughput and/or hardware can be a major factor.
Without code samples of what you're processing or knowing where your files are located though, it's hard to guess what could be improved.
-Dave Summers-
Even more Fox stuff at:
- Thread starter
- #3
Thanks Dave for your response, all of our original data resides on the newly installed veloci raptor hard drive. Also, we have exclusive access to the files. We don't use indexes and will look into that. But it seems if we are using these fast hard drives that there would be some noticeable difference in processing time. Due to the increase in write speed of the new harddrive. Also, the computer with the fast hard drive is not on a network.
Mike Lewis
Programmer
CVN75,
I don't have any specific experience of Velociraptor drives in particular, but on one of my projects, the client migrated from conventional file servers to network-attached storage (NAS) drives, and we got a huge increase in performance in our VFP apps.
I think the fact that you don't have any indexes is probably the most significant factor here. Unless your tables are very small, or you only ever process them sequentially, a lack of indexes will definitely hamper your performance.
Apart from that, I'd echo everything that Dave has said -- with one further point: the location of your executable files could make a big difference. Placing these files on each users' local drives will usually give better performance than keeping a single copy on a server.
Hope this helps.
Mike
__________________________________
Mike Lewis (Edinburgh, Scotland)
Visual FoxPro tips, advice, training, consultancy
Custom software for your business
I don't have any specific experience of Velociraptor drives in particular, but on one of my projects, the client migrated from conventional file servers to network-attached storage (NAS) drives, and we got a huge increase in performance in our VFP apps.
I think the fact that you don't have any indexes is probably the most significant factor here. Unless your tables are very small, or you only ever process them sequentially, a lack of indexes will definitely hamper your performance.
Apart from that, I'd echo everything that Dave has said -- with one further point: the location of your executable files could make a big difference. Placing these files on each users' local drives will usually give better performance than keeping a single copy on a server.
Hope this helps.
Mike
__________________________________
Mike Lewis (Edinburgh, Scotland)
Visual FoxPro tips, advice, training, consultancy
Custom software for your business
This is interesting, if I assume that your shiny new drives are not sitting in some bottleneck - on a USB connection or somesuch - then my instinct would follow Mike's.
It is VFP's management of data via the use of index files that NORMALLY makes it fly - it's probably that trick that tempted M$ to buy the technology.
Without the indexes, you need RAM to speed things up in your environment - lots of RAM can make quite a difference in an 'exclusive' world.
The snag is, most 32 bit OSs are limited to making use of 3-4GB of RAM - so that might not help, VFP is a 32 bit app, and can't make much use of the extra RAM! Except as an enormous buffer for the HDD!
If you can analyse the code and identify some indexes that might help - you might do better...
Good luck!
Regards
Griff
Keep [Smile]ing
There are 10 kinds of people in the world, those who understand binary and those who don't.
It is VFP's management of data via the use of index files that NORMALLY makes it fly - it's probably that trick that tempted M$ to buy the technology.
Without the indexes, you need RAM to speed things up in your environment - lots of RAM can make quite a difference in an 'exclusive' world.
The snag is, most 32 bit OSs are limited to making use of 3-4GB of RAM - so that might not help, VFP is a 32 bit app, and can't make much use of the extra RAM! Except as an enormous buffer for the HDD!
If you can analyse the code and identify some indexes that might help - you might do better...
Good luck!
Regards
Griff
Keep [Smile]ing
There are 10 kinds of people in the world, those who understand binary and those who don't.
Olaf Doschke
Programmer
You should determine what resources are needed before attempting to optimize hardware. It seems that you jus made a pure guess a faster HDD will help. Consider this: If your programs output written to and read from disc within this 16 hrs is not at all 16 hrs x the mix of read/write speed in bytes/s, then the hdd is not the main usage and not the limiting factor.
As an example: If your process crunches on 1GB of data it reads and outputs say 100 MB of condensed data, even assuming you need to read this amount of data in 10 times and write and overwrite it about 3 times, that'll make a load of 10GB read and 300 MB write. Even a cheap and slow HDD will do this in split parts of these 16 hrs, so the main time would not be that read/write operations anyway.
Besides that a VFP program runs single threaded and only can make use of a quad core CPU if you start some VFP EXE 4 times, each EXE processing a quarter of the data, or if you make use of VFP multithreading (
The hard drive doesn't seem to be the performance limiting factor in your processing. Even if things are read and written to DBF files directly without usage of Cursors, VFP uses caching and thus does things in memory, only flushing out to HDD when really needed.
Like the others already said indexes are of the essence and even if you only sequentially read files and process them, indexes can help a lot with that. But additional to simply creating them you also need program logic to make the best use of them, eg rather use SQL than SET FILTER and loops and rather use SEEK in indexes than LOCATE just as two starting points.
Bye, Olaf.
As an example: If your process crunches on 1GB of data it reads and outputs say 100 MB of condensed data, even assuming you need to read this amount of data in 10 times and write and overwrite it about 3 times, that'll make a load of 10GB read and 300 MB write. Even a cheap and slow HDD will do this in split parts of these 16 hrs, so the main time would not be that read/write operations anyway.
Besides that a VFP program runs single threaded and only can make use of a quad core CPU if you start some VFP EXE 4 times, each EXE processing a quarter of the data, or if you make use of VFP multithreading (
The hard drive doesn't seem to be the performance limiting factor in your processing. Even if things are read and written to DBF files directly without usage of Cursors, VFP uses caching and thus does things in memory, only flushing out to HDD when really needed.
Like the others already said indexes are of the essence and even if you only sequentially read files and process them, indexes can help a lot with that. But additional to simply creating them you also need program logic to make the best use of them, eg rather use SQL than SET FILTER and loops and rather use SEEK in indexes than LOCATE just as two starting points.
Bye, Olaf.
Olaf Doschke
Programmer
Just to give you an idea of how bad a LOCATE for a record is, when there are no index whatsoever to optimize the search:
In a table with the size of say 100 MB and the searched record being anywhere in this dbf, the DBF needs to be read from start to the record matching the search, which in average means rading in 5MB just to locate one record of perhaps about 1K size. This is a waste of time and though you would expect a twice as fast HDD do this twice as fast, once VFP has read the 100MB of the DBF into it's memory, the limiting factor is not the HDD read speed anymore.
But also within cached data reading half the DBF size in average to locate one record is a waste of time.
With an index you have a binary tree of nodes, like doing a number guessing algorithm, this typically needs log2(reccount) reads to find a record, that is in a DBF with 2^20 records (about a million records) it needs at max 20 reads within the CDX ( in average rather 19) to find the right recornd number instead of an average of 500,000 reads within the DBF (or the cached DBF in memory).
See? That's a ratio of 20:500,000, and the CDX reads are even reading less bytes in general, than a record size is.
It compares to number guessing. Guessing a number of 1-1000 by guessing 1,2,3,4,5... you end up with 500 guesses in average. If on the other hand you start with 500 and half the amount of numbers which each guess, you need a maximum of 10 guesses only.
Bye, Olaf.
In a table with the size of say 100 MB and the searched record being anywhere in this dbf, the DBF needs to be read from start to the record matching the search, which in average means rading in 5MB just to locate one record of perhaps about 1K size. This is a waste of time and though you would expect a twice as fast HDD do this twice as fast, once VFP has read the 100MB of the DBF into it's memory, the limiting factor is not the HDD read speed anymore.
But also within cached data reading half the DBF size in average to locate one record is a waste of time.
With an index you have a binary tree of nodes, like doing a number guessing algorithm, this typically needs log2(reccount) reads to find a record, that is in a DBF with 2^20 records (about a million records) it needs at max 20 reads within the CDX ( in average rather 19) to find the right recornd number instead of an average of 500,000 reads within the DBF (or the cached DBF in memory).
See? That's a ratio of 20:500,000, and the CDX reads are even reading less bytes in general, than a record size is.
It compares to number guessing. Guessing a number of 1-1000 by guessing 1,2,3,4,5... you end up with 500 guesses in average. If on the other hand you start with 500 and half the amount of numbers which each guess, you need a maximum of 10 guesses only.
Bye, Olaf.
Does your server have one drive or set of drives/controller for the Windows Server OS (Drive C
and a separate set of drives/controller for the data (Drive D:, E:, etc). If not, that could be a potential bottleneck.
I didn't handle the conversion, but when there was a server migration from a server with all logical (partition) drives sharing one set of RAID drives to a server with separate sets (one RAID set for OS, one RAID set for data), there was a marked 5-fold speed increase in the processing speed. There may have been other differences, but that's the one I know about.
and a separate set of drives/controller for the data (Drive D:, E:, etc). If not, that could be a potential bottleneck.I didn't handle the conversion, but when there was a server migration from a server with all logical (partition) drives sharing one set of RAID drives to a server with separate sets (one RAID set for OS, one RAID set for data), there was a marked 5-fold speed increase in the processing speed. There may have been other differences, but that's the one I know about.
- Status
Similar threads
