Performance improvement tips required

[mlapse]

Hi,

I face an issue with the processing performance of my mercator systems.

We are using Windows NT 3GB memory and 3 GhZ processor. However sometimes files seem to get backed up a lot.

We receive a lot of file over the day.. almost 100000 in a day. THe size of the files could also vary from 2 KB to 10 MB.

I tried playing with the InitPendingHigh and Low in the ini fule and it seemed to help a bit. I set the High at 1000 ans low at 200. However this helped in clearing the backlogs but it doesnt seem to process the files in the order they entered the system i.e. though the files get processed faster, some files may be processed right at the end and so those may have waited a long time! At any given point in time there would be about 10000 files in the system. Out of these about 8000 will belong to a particular partner which can take some time to process. However other data may be time sensitive and needs to be processed fast.

Are there other settings which can be used. The MaxTHreads is set to 20 currently. We have a total of about 50 systems running and about 200 maps on this server.

How can I improve the performance of the system on the whole?

 

[janhes]

Put the files that ned to be processed quickly in a different folder or give them a different name. Then have a second system that is trigered from the new location/file name.

 

[janhes]

What is the max concurrent maps set to in the .msl? Can you have multiple copies of the map running?

 

[BocaBurger]

What version of Mercator are you running? If you really have NT, you might consider upgrading, if your Mercator version supports it. If you only have one CPU, reduce maxthreads to between 8 and 12. Thread switching does reduce performance.

However, it might be that you need more hardware. The priority settings in the IFD don't work. Priority maps need to have the maxconcurrent map instaces setting in the IFD set to 3 or 4, so that they always have some threads available even if the rest of the system is busy. Increasing the number of running threads slows down the systems. You would need to add CPUs and upgrade your license. You might want to get 8.0, so that you can use the map optimizing tools now available.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[mlapse]

Janhes,

We already put the files for each partner in separate folders and run diff system from there.

The max concurrent map instances is 20 which is the same as MaxThreads.

Bocaburger,

I dont think we can upgrade just yet. What is the diff between map instances and concurrent maps? Is it that if the first is 20 and the secon is also 20, in effect I could have 400 maps running together and processing 400 messages at a time?

 

[BocaBurger]

mlapse,

Maximum concerrent map instances is a separate setting for each map in the Intergration Flow Designer. Maxthreads is a configuration setting in the .ini file. Unless you have a 5 or 6 CPU server, it could be your maxthreads is too high.

What version of the product are you running one? Are you really still using NT? Note that when an OS is sunsetted by the publisher, any version of Mercator / DataStage TX is no longer supported on that OS.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[mlapse]

Bocaburger,

You were correct. We are using Windows 2000 server.

Can you provide some specific tips for performance improvement while using this OS.

 

[BocaBurger]

I gave some already. Reduce maxthreads. Tune the maps. Tip: Even if you are not ready to go to 8.0, get it and make a copy of your maps. Use 8.0's map profiler and paging setter to optimize the maps. Review your launcher / system.txt file for conflicts. Use run map caching. Use streammaxmemlimit for large data files, set to 15 to 25 max. If using MQ, uncomment and set poll min to 25.
Make sure all RUN maps are multi-thread capable and make sure no RUN command breaks this.
Do not turn on any adapter or input tracing. Do not turn on unique audit logs for every map.

Any more and I will send you a bill



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!