Window freezing

[138006]

Hi,
I have a problem -- let me try to explain.

I have 2 compare data of 2 different databases using PERL. Now that is happening correctly. To show a progress bar I am using TK.
Now, when the number of records of the 2 tables (which I want to compare) is huge, then the TK frontend window gets completely blank if I do a "ALT-TAB" for some other window. E.g. I run my comparison tool by clicking on the "Start" Button on my TK window. Then do a "ALT-TAB" and write in a word document. Then again when I do a "ALT-TAB" to see the progress of comparison I find the TK screen to be completely blank. SO, in effect, there is no meaning of the progress bar feature if I can't see the progress!!

If I don't do "ALT-TAB" and keep the TK window active only, then I can see the progress.

Now this might be a memory problem of Windows(I am running on Win2000)--however this problem is not occuring when I run other applications and this is the concern.

Can I do some tweaking in the TK code (the PERL portion for comparison is fine) to make the memory usage more efficient so that it can paint the TK windows after the ALT-TAB??

Please suggest the best practices regarding this. Since memory handling is not there explicitly in TK I don't know what to do.

Please help.
Thanks.

 

[138006]

Not Really.
I just wanted to get some affirmitive nod from someone that looking in these areas may give some lead to my problem. If these are areas that are absolutely non-related to my proble, then I will not spend time on them.

This is the basic objective

 

[icrf]

Your database call is taking a long time? The DoOneEvent() trick only really works when your lengthy processing is done in some kind of loop in Perl. If a database call is taking a long time, the $sth->execute() (or whatever) isn't returning until the database is done. Perl can't handle any events until the DB call is done b/c it's waiting for that return code before moving on.

While Tk may not be thread safe, you should be able to fork() off a process (forked processes don't share memory [maybe just file handles], right?). Then you'd have to use some kind of explicit IPC (Inter-Process Communication) for the forked process to return that it's completed.

I've never used a progress bar, but I'd imagine it needs some way to be told to update. How do you do that if all your time is really in external database calls that Perl can't track? If you have several DB calls, it can, but if each call is taking a significant amount of time, maybe some rethinking of how the whole process works needs to be done.

Anyway, hope I've been of some help. I'll try and keep an eye on this thread and maybe do some testing tonight.

________________________________________
Andrew

I work for a gift card company!

 

[MikeLacey]

fork() and Tk go together like oil & water - in my limited experience.

You want unexplained crashes and errors? You'd like s/w that fails at unpredictable points in exotic ways? The fork() and Tk are for you...

Mike

"Deliver me from the bane of civilised life; teddy bear envy."

Want to get great answers to your Tek-Tips questions? Have a look at faq219-2884

 

[PaulTEG]

if there's an issue, it needs to be solved ... neh?
I'd be inclined to go icrf's way of thinking ... until Mike said twasn't a good idea

To me it looks like an opportunity for someone with a common goal (I've used this statement in the past ... and got away with it), to assist the Tk community in finding why the 'unexplained' and 'exotic' issues exist, and what can be done to resolve the issues.

If you can recreate a problem at will, then there's something to look at ... consider posting to the bugs area for the specific module (they'll need your code and data to recreate, not your entire project, just what you've whittled down to being the problem), and asking the actual developers for assistance. Once they've seen you've made an effort to make it recreatable for them to identify, then there's a chance they'll be more inclined to suss out the issue ...

Just my €0.02

--Paul

Nancy Griffith - songstress extraordinaire,
and composer of the snipers anthem "From a distance ...

 

[138006]

HI All,
Thanks for the suggestions.

icrf, for ur better understanding let me explain the situation.

I have a .txt file whose contents are like this below:
a1,b1
a2,b2
a3,b3
a4,b4
etc

where a1,a2,a3,a4 are columns of table T1 of Database1 and b1,b2,b3,b4 are columns of table T1 of Database2. key1 and key2 are the 2 join keys between the 2 tables

I need to compare data in these 2 tables using PERL.

What I have done is for all rows of this file I run a sql in a loop (one by one for each column pair) that counts the number of match and mismatch records. The query is something like this:

SELECT SUM (CASE WHEN Schema1.T1.a1 = Schema2.T1.b1 OR (Schema1.T1.a1 IS NULL AND Schema2.T1.b1 IS NULL) THEN 1 END) AS MATCH,
SUM (CASE WHEN (Schema1.T1.a1 <> Schema2.T1.b1) 
OR (Schema1.T1.a1 IS NULL AND Schema2.T1.b1 IS NOT NULL)
OR (Schema1.T1.a1 IS NOT NULL AND Schema2.T1.b1 IS NULL) 
OR (Schema1.T1.key1 IS NULL AND Schema2.T1.key1 IS NOT NULL)
OR (Schema1.T1.key1 IS NOT NULL AND Schema2.T1.key1 IS NULL)
THEN 1 END) AS MISMATCH
FROM Schema1.T1 FULL OUTER JOIN Schema2.T1 ON Schema1.T1.key1 = Schema2.T1.key1

AFter each row is complete I store data in an array and finally when all rows are done I write the output in a file.

Now this is the query that runs in my DB2 database --this is taking a long time (the timne taken by it is correct and conforming with the volume of data).

In the front end I have TK that apart from some other buttons for selecting the source file, target file, configuration files etc, also starts the query and shows a progress bar. Now if number of columns is around 15-20 the tool runs fine and quickly also. But it just hanges if I take around 150 column pairs.

Your suggestions....

 

[138006]

Hi,
What i did is finally:

Have a progress bar that I create while entering the loop where i run the SQL as mentioned above. And then destroy it just before going out.

In this way I can keep the progress bar active. However I can't bring it up while I do Alt Tab. It resurfaces from the tray when I click on it and that too when all other windows are minimized. How can I keep it active even when users do ALT TAB to refer other applications and then come back to it?

Regards,

 

[138006]

Hi All,
Still there is some problem. It is working as mentioned above for my .1 M records but not responding for 270M rows.
Please note that there is FULL Outer Join in the query so its understandable that there is lot of work done in the query --basically Merge Join sort of thing taking place between 2 tables with 270M rows in each !!!

Is this time being very long the reason while after the query is over, there is no information passed to TK (thats what it seems) and TK cannot refresh its screen?

Any help reagrding this volume testing part will be appreciated.
Thanks

 

[mlibeson]

I have had the same problem you have described. I modified my code to work a little differently then I initially intended. I have include below, the code snipets from my program where it interacts with my progress bar so that you can compare. The task that the progress bar is keeping track of, I call from a subroutine when I create the taskbar.

=CODE
#####
##### PROGRESS WINDOW
#####
$main::amt_done = 0;
$main:ercent_complete = 0;
$main::datafile = "";
$main::mw = MainWindow->new;
$main::mw->geometry("${main::wmwidth}x80+$main::wmpos");
$main::mw->minsize(${main::wmwidth},'80');
$main::mw->maxsize(${main::wmwidth},'80');
$main::mw->title("$main:rogName [Status]");
$main::r0 = $main::mw->Frame;
$main::r0->pack(-side => 'top');
$main::r1 = $main::mw->Frame;
$main::r1->pack(-side => 'top');
$main::r2 = $main::mw->Frame;
$main::r2->pack(-side => 'top');
$main::r3 = $main::mw->Frame;
$main::r3->pack(-side => 'top');

$main::r0->Label(-width => '80', -text, &Win32::GetCwd(), -font => "$main::tfont")->pack(-side => 'left');

$main::r1->Label(-width => '40', -text, "Files Processed", -font => "$main::tfont")->pack(-side => 'left');
$main::r1->Label(-width => '1', -text, '%', -font => "$main::tfont")->pack(-side => 'right');
$main::r1->Label(-width => '5', -textvariable, \$main:ercent_complete, -font => "$main::tfont")->pack(-side => 'right');
$main:rogress1 = $main::r1->ProgressBar(
-width => 20,
-borderwidth => 2,
-padx => 2,
-pady => 2,
-relief => 'sunken',
-from => 0,
-gap => 0,
-to => 100,
-blocks => 100,
-colors => [0 => 'blue', 25 => 'green' , 50 => 'yellow', => 75 => 'red'],
-value => $main:ercent_complete,
-variable => \$main:ercent_complete
)->pack(-side => 'right', -fill => 'x');

$main::fne = $main::r2->Entry(-width => '40', -text => \$main::datafile, -font => "$main::tfont")->pack(-side => 'left');
$main::r2->Label(-width => '1', -text, '%', -font => "$main::tfont")->pack(-side => 'right');
$main::r2->Label(-width => '5', -textvariable, \$main::amt_done, -font => "$main::tfont")->pack(-side => 'right');
$main:rogress2 = $main::r2->ProgressBar(
-width => 20,
-borderwidth => 2,
-padx => 2,
-pady => 2,
-relief => 'sunken',
-from => 0,
-gap => 0,
-to => 100,
-blocks => 100,
-colors => [0 => 'blue', 25 => 'green' , 50 => 'yellow', => 75 => 'red'],
-value => $main::amt_done,
-variable => \$main::amt_done
)->pack(-side => 'right', -fill => 'x');

$main::r3label = $main::r3->Label(-width => '40', -text, "Press the \"s\" key to begin.", -font => "$main::tfont")->pack(-side => 'left');

$main::mw->bind('<Control-c>' => sub{print STDOUT "Cancelled program.\n";writeln("ERR", "Cancelled program at ", &timestamp, "\n");exit(0);});
$main::mw->bind('<KeyPress-s>' => [\&mainblock]);

MainLoop; ##############
=cut

=Within the subroutine call
$main::file_count++;

$main:ercent_complete = sprintf("%6.2f", (($main::file_count/(scalar(@main::logfiles))) * 100));
$main::mw->update();
=cut

=After the subroutine call
###
### END OF PROCESS CLOSE PROGRESS BAR WINDOW.
###
$main::mw->destroy();

=cut

Within the subroutine call I call another subroutine that also updates the progress bar as follows:

=code
######
###### PROGRESS
######
$main::amt_done = sprintf("%6.2f", ((($main:laceholder / $main::eofpos) * 100)));
if ($main::amt_done <= 0) {
$main:ercent_complete = sprintf("%6.2f", (($main::file_count/(scalar(@main::logfiles))) * 100));
} else {
$main:ercent_complete = sprintf("%6.2f", ((($main::file_count/(scalar(@main::logfiles))) * 100) + ((1 / ((100/$main::amt_done) * scalar(@main::logfiles))) * 100)));
}
$main::mw->update();

=cut

I hope this helps you.

 

[138006]

Hi,
Thanks for the code. What was the nature of your background job? Was it a database query with so huge data. It seems all is lost due to this huge data volume.

My code is running fine on a small vol of data.

 

[mlibeson]

I was processing large data sets from flat files and loading them into a database. I use the program for part of a process for web traffic auditing.