Estimating Timing for a loop

[Bryan - Gendev]

In the first step in my app I recombine a number of related tables into one large table with a lot of processing to determine the content of some of my special fields based on items in one or more of the tables. I do this in a Scan /Endscan of the main table containing the 'hooks' to the related tables.

As this process can take quite a while I would like to estimate the potential time the loop might take, much like Windows copy dialogues and others show.

It seems to me that I could measure the time taken for, say 10 records, and extrapolate for the last 90%. I do use a progress bar but with a large number of records its initial progress is minimal. Perhaps I could test after 20% and adjust the estimate.

Has anyone done anything like this or is there some code snippett somewhere to help me on my way?

Thanks in anticipation.

GenDev

 

[Mike Lewis]

GenDev,

I suggest that you forget about the times, and instead base your progress bar on the number of times round the SCAN loop.

You know the number of records in the table being scanned. You also know the record number of the current record at any one time. Those two numbers will give you the percentage complete:

lnPercent = (RECNO() / RECCOUNT()) * 100

I know that this is not an accurate estimate of the time to completion. Some records might take longer to process than others. But the point is that it gives the user a rough idea of how fast the process is going, and reassures them that the program is not hanging.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[Bryan - Gendev]

Thanks Mike,

I'm already showing them the percentage in the progress bar but as I said the early movement in a 'large' project is pretty slow and my user still wanted to know how long it might take...

 

[Mike Lewis]

In that case, I'd go with your idea of timing the first few records, and extrapolating from there. Better still, do the extrapolation each time round the loop, so as to constantly refine it.

Better still, do both. Show the percentage complete, and also the estimated time to completion.

But, for the latter, don't be too precise. If it's more than five minutes, show it to the nearest minute. If it's one to five minutes, show it to the nearest ten seconds, and so on.

You can use the SECONDS() function to record the start time, and then use it again repeatedly to measure the time that has elapsed so far. Multiply that by the record count and divide by the current record number. That will give you the total time. Subtract the time so far to give the time to completion. (Sorry, I expect you already knew all that.)

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[Bryan - Gendev]

Thanks Mike,

Good to have the idea confirmed...

 

[TamarGranor]

GenDev - I just want to point in one other direction. Are you sure that your process is as fast as it can be? If you could speed up the processing, a more detailed progress bar might not be needed.

Why don't you tell us a little about your set-up and what you're doing and maybe we can point to speed-ups.

Tamar

 

[JRB-Bldr]

Also, while Mike's suggestion is a good one, you might want to periodically double-check the time estimate based on the records/sec (or something like that) as you process the records.

For example when you download a file across the net the download routine often gives an estimate on how fast the bytes are arriving. Since that value often changes, the estimate is periodically and automatically recalculated to reflect the current download speed.

If network traffic, virus checking, etc. were to impact your processing speed during the processing, it would affect the time it would take to accomplish the task. Periodically double-checking and re-calculating the remaining time might be appreciated by your users.

However Tamar does indeed have a valid point.
Perhaps your processing could be sped up a good bit and thereby minimize the need for an 'estimated time'.

Good Luck,
JRB-Bldr

 

[Mike Lewis]

One other point: the act of reporting progress might itself slow down the process you are reporting.

In general, a screen update is a relatively slow operation. If you update the screen (that is, the progress display) from within a tight loop, the update itself might have a disproportionate effect on the running time.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[Bryan - Gendev]

Thanks everyone..

In looking more closely I actually have 2 progress bars around 2 separate processes.

In my measurements I find

Process 1 - less than a minute with 'manually' supplied approximate percentages shown in which I open 20plus 3rd party tables and do some counting.

Process 2 - 5 minutes - in a Scan/Endscan loop. This process takes a single 3rd party table and amongst other things works out the filesize for a referenced file stored in each record in the table. Each file can be anywhere on a users system. I record these values and use them later in my program to save doing it all again.

lncount= Adir(laTemp, myfile )

If lncount> 0

 imagebytes =  laTemp(2)    && file size in bytes
 imagesize = Str(imagebytes/1024,6,1)+ 'K'

endif

I have shown some of the comments offered by the respondents to my user and am awaiting his response.


GenDev

 

[Olaf Doschke]

I'd either show 2 progressbars or a combined form with 2 progress bars. As you say you know the approximate times you can label the bars with them and then compute an estimate while each process runs.

For a file copy or such process it's of course better to estimate by the amount of bytes copied in relation the the sum of all file sizes. In general it's best to have a measurement rising linear in time to estimate the total time of course.

In one case of copying a large folder to USB drive I use robocopy and in the first pass let it just create a log file without copying and then compute the total size of files it would copy. In the second pass I let robocopy log again and then can see how far it is in bytes.

As it's a secondary process, the foxpro application just has to show the progress. The users can work on anything else, while the robocopy progressbar is drawn by a timer in the status bar.

Bye, Olaf.

 

[TamarGranor]

Process 2 - 5 minutes - in a Scan/Endscan loop. This process takes a single 3rd party table and amongst other things works out the filesize for a referenced file stored in each record in the table. Each file can be anywhere on a users system. I record these values and use them later in my program to save doing it all again.

Do you really need all the sizes at this point and later on? Might it make more sense to find them on an as-needed basis, since you might only need a few of them?

I suspect you're right that this is the slow code. You might test that by checking the time before and after the loop. If that's not the slow part, then test some other sections to see where the slow part is.

Tamar

 

[Bryan - Gendev]

Tamar,

The sizes are needed to produce a catalogue of the image and other files in the 'project' so I produce a composite table to run the report.

GenDev

 

[TamarGranor]

Seems to me one way you could cut down the time here would be to organize the list, so that you could (theoretically) use a single ADIR() to pull data for multiple files. You'd still have a worst case scenario where every file is in a different folder, but I suspect that far more often, you're really looking at only a few different folders.

Try doing a query on the data, including JUSTPATH(myfile) and ordering on that field. Then, grab that list, and use ADIR() only once on each folder.

Tamar

 

[Bryan - Gendev]

Thanks Tamar,

Unfortunately the file path is in any of 5 fields in the original table.

Trying to get them into a composite table may take a while. I'll try it.

GenDev