InitPendingHi & Low 1

[clk430]

Scenerio:

The app area drops 5,000 files all at once in a trigger directory. The map that pulls these files are wildcarded, however it is single threaded. This takes a whole day to process all of these files and FTP them to the TP.

We have 50 other systems with legitiamte watches/triggers that need to be satisfied throughout the day.

What are some InitPendingHi & Low settings that I can set that will alleviate the monoploization of the 5,000 file system? Currently, they are 0.

 

[eyetry]

clk430, if I remember correctly you are running on a 1 cpu windows machine..... Assuming that's true, you're going to run into diminishing returns multithreading.

It takes longer for the CPU, memory and disk to swap back and forth between processes than just handling requests sequentially. With the volume you process you see that slow down pretty fast.

You should be able set the priority on your systems so that the most important jobs run sooner. If some ofthe 50 other systems are of higher priority than the FTPs....

also, if some of the FTP files are higher priority than others and there's some distintion in their file names you can set priority accordingly.

 

[dstxaix]

One thing you need to consider is the number of files in your trigger directory. The more the number of files in the directory, the longer it will take to pick the file from the directory. All Operating Systems (excluding BSD) have this problem. My suggestion would be to write a script to throttle the files into your trigger directory. Make sure the number of files never goes above 1000 or 2000 files or whatever number works best for you.

 

[clk430]

I see guys.

OK, assume I'm not going to multi-thread teh app and keep it as it is. When a 1000 files get dropped in the trigger folder, 1000 init pending maps occur. Will playing with the InitPendingHi & Low stop this from happening, and ultimately slightly improve performance? What are the negative ramifications?

 

[BocaBurger]

You could multi-thread the map, and set maxconcurrent map instances to 2 or 3. Depends on where the bottleneck of the FTP is located.

Would need to see what the map does (mapping/ functions etc) and so forth. Init pendings do not consume resources like resource pending does.

Initpendinghigh setting just stops the scanning when the number is reached and resumes when the low occurs. This will slightly improve performance. Tuning page size and count, maxthreads, initpendingidle and other settings can also help.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

Resource pending? That is a map waiting for a resource to free up before executing, right? Init is just the trigger being satisfied.

 

[BocaBurger]

And the map is not yet running.


BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

Why can't the $*%^# pdf manual explain it as easily as you do??

What will initpendingidle buy me?

 

[BocaBurger]

If the manual was perfect, I wouldn't have a job

Initpendingidle reduces CPU cyles. If you have maps in init pending, you don't want to spend too many CPU cycles checking to see if you can run the map.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

Job security, so that's the motive... lol That was funny.

What would be a good, conservitive setting? 1 proc, 2.8ghz, 2gb ram, 500+ watches, 50msl, 200mms. etc. The reason I ask is becasue we are currently pegged at 100% (CPU usuage) 95% of the time. Increasing listensleepcount to 500 helped some.

 

[BocaBurger]

Well, in reality, 500+ watches is a bit much for 1 CPU. You are spending most of the time watching. Initpendingidle could be 100 to 500.
In reality, a 500+ watch system should have 4 CPUs min and 4 gig RAM, min., depending on what the maps do when they are triggered. Just file to file, text blob with no transforms can be a lot more maps than EDI to XML and back or XML to SAP etc. If one map takes 100% of a CPU while it is running, then with maxthreads set to 4, it will take about 4 times longer to process 4 maps. Either 4 of the same or 4 different maps.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

Huh, it's already been set at 1000. Is that too much?

 

[janhes]

yes

 

[BocaBurger]

COuld be. 1000 = one second. This could be a bit long for a computer. 400 to 500 might work somewhat better but still keep the CPU usage down.
You have to tune the various variables (one or two at a time) to get the best performance and useablity.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

Boca,

I've tuned the .ini per our discussion (in this, other threads and IBM email support - (i think I know who you are)) and have had EXCELLENT results.

So far, I've changed:
ListenSleepTime from 50 to 500
InitPendingHigh from 0 (unlimited) to 200
InitPendingLow from 0 (unlimited) to 100
WatchMaxThreads from 20 to 8
MaxThreads from 20 to 12
RunMapCaching from ;300 (disabled) to 100
initpendingIdleMS from 1000 to 500

These changes have sped up our systems 30% faster with 50% less errors. You are the man! Of course, these arew baby steps and will need to be re-tuned, but so far, so good.

Now I've gotten greedy. I want to change more! Specifcally, increase filelistencount. It is at 25 (default), what value would be appropritate?

Here is the catch though. I've worked extensivlely with our hardware people and they say that we have 1 CPU. Documentation says we have 2. Furthur investigation led to the discovery that we have 2, but only one is enabled. Very frustrating. I adjusted the maxthreads to 4 and had terrible results. Setting them to 12 vs. 20 yeilded great results. So, I assume we are using two.

ANYWAY, with the 500+ watches, what would you recommend for filelistenSleep?

And considering we have 2 proc "now", what values of the setting that are set now can I "test" with?

I wish this forum had an evaluation to fill out so I could light it up!

 

[BocaBurger]

If you are on Windows, open Windows Task Manager, if there are two CPUs it would show. You might have one hyperthreading CPU and that can be turned on or off. I don't know how you can turn CPUs on or off in a Wintel machine.

Just test some numbers with the variable you want to try. Basically you want to see if the CPU utilization changes (lower is better) when the system is idle.

Now you have to tune the maps themselves. Get 8.0 and work on copies and transfer the best settings for page size & count to your source system. Use the profiler to improve map performance and copy the rules to your source.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[JEMIAZ]

For Wintel machines to disable a cpu we use the following in the BOOT.ini

multi(0)disk(0)rdisk(0)partition(2)\WINNT="ONECPU Microsoft Windows 2000 Advanced Server" /fastdetect /onecpu /pae

/onecpu is the command
/pae encourages the server to use memory above the 4GB limit

Beware that hyper threaded processors like XEON show in the task manager as 2 CPUs - ie on a 2 CPU box will show as 4.

Changes to Boot.ini are made at your own risk.

Tim

 

[BocaBurger]

Ah ha. Advanced Server. I only get to play with the regular stuff
The /pae is good to have, if you installed the RAM.
Why would you want to disable a CPU?




BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

Boca, guess what? I finally got an answer. We have 1 2.8GHZ CPU (not 2 as I thought) however it is hyper-threaded.

The current maxthreads and watchmaxthreads are 12 and 8, respectively.

Does a hyper-threaded CPU simulate two physical CPU's? Or Can I lower my maxthreads?

 

[BocaBurger]

From my personal limited testing, a hyperthreaded CPU is not twice as fast as a single CPU. Your settings would need to be determined by your own testing. Have Task Manager open and load test your server. If you spend most of the time at 100% of both CPUs, then you might get better performance at a lower number. If you don't hit 100% on both, add a thread until you are using about 100% of one and 85% of the other. This sould be at a load of about 125% of your expected peak data volume, near the end of the server's proposed lifetime. (If you get the latest / greatest today, it will be technologically obsolete in 5 years.) This does not mean the machine is obsolete, it just should be used for a less intensive task, since in 5 years the current machine at the same price would far outdistance the old machine, assuming things do not change in the CPU world.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!