Multithreading of Mercator Maps

[mlapse]

I have an issue which needs fixing. There is a system I have built which should enable upto 20 instances of the maps to run simultaneously.

However when the input files are about 50000, the system seems to process files one by one.

Why is this happening and how can I overcome this?

 

[eyetry]

I don't recall what version you are using or your platform….. Also curious, how many CPUs on the box?….

I may remember this wrong, its been a while since we looked at multi-threading. At the time we had one CPU license and multi-threading reduced performance. We are running a W2K Server. I ran grinder scripts against the event several times as synchronous and then asynchronous. The intent was to maximize realtime transaction performance.

What I recall is that to multi-thread you need to turn off tracing. Log, output and backup file names have to be unique (or off) or the events would step on themselves. The EventServer will not allow this to happen. You'll need to use the same rules for any runmaps being executed too. Seems like I tweaked the Mercator.ini file too. I don't have my notes on it handy, but if your maps use adapters like DB, HTTP, EMAIL, FTP etc… you may need to confirm related settings in the ini file allow for enough concurrent connections.

Jujutsu? I know tweaked some of the ini file settings but can't for the life of me remember which ones I played with. Do you have any recomendations?

 

[janhes]

Have you set the InitPendingHigh=20 in the mercator.ini file?
I have found that with a lot of pending files the system grinds to a halt.

 

[BocaBurger]

What version of Mercator, on what platform? There could be a lot of settings needed to get 50k files dropped at the same time through your system. Could also be that the server is not big enough to handle this. How big are the files? How many maps does it take to completely process each file (RUN maps and maps triggered from the initial map)?

Best bet is to send mercator.ini (or dstx.ini) and launcher.txt (compoundsystem.txt) to Support for analysis. For some reason it won't let me log in as Jujutsu. I had to create that log in when this one stopped working. Not the $%#%$#%)(* system won't let me change the e-mail for Bocaburger, since it is in use by Jujutsu log in, and I can't change that since it won't let me log in. System admin doesn't respond.


BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[mlapse]

Hi,

I am using Windows NT as the platform and Mercator version 6.7 (release 302).

The files dont seem to go through the system fast enough. There is just one map in the system which should pick up the input files and process them.

What is the diff between the Fetch As set to Integral and Burst. Would it make a diff at all.

 

[eyetry]

CPUs? Memory? Are you using static data/log file names?

Rereading your earlier post.... as of now, you can't run anything simultaiously? or is the problem getting 50k files to run simultaiously?

 

[mlapse]

thanks for helping eyetry.

the CPU is 3.8 Ghz and the memory is 3 GB.

No i have turned off the log files and the trace files.

initially when i had the workspace set to "file", the system was processing just one file at a time. on changing it to memory, multiple files are now being processed.

however when i have 50K files in the input, it takes a very very long time to process. the files are all 4KB size.

I want that this system should be able to process all these files fast.

Does the Fetch As "Burst" option change anything?

 

[janhes]

What have you got set in your .ini file for InitPendingHigh?

 

[mlapse]

jahnes - i have the value as the default of 0 which means unlimited.

Are you saying that with more files, mercator needs to keep track of each of the watches and therefore slows down?

 

[janhes]

Yes - it spends all of it's time setting up triggers and has no resource left to process the triggers. Try changing this to InitPendingHigh=20. That should solve the problem. Also have MaxThreads=20

 

[eyetry]

...however when i have 50K files in the input, it takes a very very long time to process. the files are all 4KB size...
"

I noted that on our machine, 1 cpu (3.4 ghz) 4gb ram processing 1-kb files took longer multi-threading than they did sequentially.

I used a grinder script and processed 25 cylcles with 100 files per cycle, 1 second between cycles. I set it up to run multi-thread and then sequential. The system processed the sequential series faster.

Problem is shared CPU and memory. The fact that you are using memory for workspace should help but it still takes resources to swap between processes.

Probably preaching to the choir so bear with me. As you know, Windows doesn't really process two items at exactly the same time. It swaps back and forth between files/processes. This happens pretty fast so you can't really tell much diff between multi-threading several small files but at the scale you're talking about the additional milisecond here and there adds up. Then, add to that, if you consume all of your physical memory the machine starts using virtual memory (disk). Things will slow down even more.

Also, keep in mind that your input files are only 4kb each but depending on map design you could be consuming a much larger chunk of memory. I've seen maps that have consumed more than 4 times the input file size(s) in resources. Are you reading other inputs? Connecting to DBs? and so on?

Is 50K files a realistic test for your environment?

eyetry,

ps: sorry, don't really feel I'm being much help here.

 

[mlapse]

jahnes,

your solution seemed to work. i made InitPendingHigh = 200 and Low=50 and it seemed to be working better.

However the only thing was that that when i made it 1000 and low 200, it seemed to be working better. Any idea why?

Also I currently have maxthreads = 20. Can I raise this value and expect maps to process stuff faster??

eyetry, thank you for your help. if there is anything else you come across, please let me know. i need to improve the performance on my system.

 

[BocaBurger]

Thread switching is overhead. See how much CPU you have left, then if 0, reduce maxthreads. You will find that somewhere between 2 and 6 threads per CPU will give the best performance.

BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!