Urgent : Source Event getting Skipped on 6.7.

[snehareddy]

Hello,


We are facing a critical issue which is described in detail below:

Currently we have five installations of event servers running on Mercator 6.7 on our production system. All installations are on one UNIX box which is running HP-Ux 11.11. The system is live since September, 2004. The event servers are, on average, restarted every 15 days.


Our design is as follows:

The input data (File/MQ message) is recieved by a system called as Router.

Router calls the actual map (called interface map) based upon the polling

folder/queue and passes the input data to it alongwith required parameters.

The audit log file is created for both Router and the actual interface map.

These log files are then picked up by another system which updates the key

fields in an Oracle database.


Recently we've faced a strange issue on one of the event servers (two times

in last 10days).

Following are the observations:



1. The audit log files created by Router component is not in correct format

. This happens randomly, when it starts, and few files are created in

correct format and others incorrect. Sample of files in incorrect format is

as follows :



<ExecutionSummary MapStatus="Valid" mapreturn="0" ElapsedSec="0.141" BurstRestartCount="0">
<Message>Map completed successfully</Message>
<CommandLine>/merccesar/maps/router/outGrA.hp</CommandLine>

<SourceReport card="1" adapter="R/3 ALE" bytes="75997" adapterreturn="0">
<Message>Succ</Message>
<Settings>-a CES_MERCATOR_400_1 -g 53.71.198.77 -x sapgw02 -ar3+ /merccesar/data/router/outGrA/log/ar3.log -bix -TID 3547C65051B043328E0444A7 -lsn 60 -event 18a803</Settings>
<TimeStamp>12:59:57 September 22, 2005</TimeStamp>
</SourceReport>

<TargetReport card="1" adapter="Sink" bytes="14" adapterreturn="0">
<Message>Success</Message>
<TimeStamp>12:59:57 September 22, 2005</TimeStamp>
</TargetReport>

<TargetReport card="2" adapter="Sink" bytes="295" adapterreturn="0">
<Message>Success</Message>
<TimeStamp>12:59:57 September 22, 2005</TimeStamp>
</TargetReport>

<TargetReport card="3" adapter="Sink" bytes="1089" adapterreturn="0">
<Message>Success</Message>
<TimeStamp>12:59:57 September 22, 2005</TimeStamp>
</TargetReport>

<TargetReport card="4" adapter="Sink" bytes="658" adapterreturn="0">
<Message>Success</Message>
<TimeStamp>12:59:57 September 22, 2005</TimeStamp>
</TargetReport>

<TargetReport card="5" adapter="Sink" bytes="0" adapterreturn="0">
<Message>Success</Message>
<TimeStamp>12:59:57 September 22, 2005</TimeStamp>
</TargetReport>

**************************************************************************


2. The component which picks log files and updates into database (running

on second event server) takes very long time (about 8-10mins) to process

each such file resulting in 2000-3000 files waiting to be picked up.


3. The most critical issue with all this is after the time when the first

of such incorrect format audit log file is created, the event server is not

able to pick up any files i.e. the incoming file is not able to trigger the

system (with Source Event ON). The event servers does not respond at all to

any new incoming file. This has been observed only for one of the event

server and also only for systems using FILE adapter as input. Systems using

MQ adapter under same event servers runs fine.



4. After the restart of the event server all the files which were lying in

their respective folders (not picked up earlier) are successfully processed

. Also the audit log files created by Router component are in correct

format.


Our system details are as follows :

OS: HP-Ux 11.11

Java: Version 1.3.1.12

SAP: 4.6c

------------------------ mercver ---------------------------------------

/opt/mercator_p/instance1/bin/mercator

--> 6.7.1(306) <-- Thu Jun 26 06:38:13 2003 - 368640 bytes



/opt/mercator_p/instance1/bin/mtsmaker

--> 6.7.1(306) <-- Thu Jun 26 06:14:51 2003 - 122880 bytes



/opt/mercator_p/instance1/bin/launcher_671

--> 6.7.1(306) <-- Thu Jun 26 06:44:01 2003 - 638976 bytes



/opt/mercator_p/instance1/bin/launcher

--> 6.7.1(306_1EM) <-- Fri Sep 10 11:08:09 2004 - 15 bytes



/opt/mercator_p/instance1/bin/mercver

--> 6.7.1(306) <-- Thu Jun 26 06:25:39 2003 - 36864 bytes



/opt/mercator_p/instance1/bin/launcher_387552

--> 6.7.1(306_1) <-- Thu Aug 5 16:17:08 2004 - 634880 bytes



/opt/mercator_p/instance1/bin/launcher_398917

--> 6.7.1(306_1EM) <-- Fri Sep 10 09:51:18 2004 - 634880 bytes

Request you to kindly inform

1. Has anyone faced a similar issue, if yes.
2. What are the causes for the same and
3. Solutions if any to solve this to this problem

Thanks in advance.

Regards,
Sneha

 

[janhes]

We run a similar setup under Windows. The only time I have problems is with JMS where the Event Server has a memory leak and if we don't reload the Event Server regularly we get all sorts of spurious JMS connection errors. Reloading corrects this.

Have you had a significant increase in the number of files processed or any other increase through that Event Server?

It sounds like a memory issue if nothing else in the software has changed.

 

[snehareddy]

Hello ,

Thanks for the update .
Yes we had a significant increase in the files processed the system is live from Sept 2004.
We observed this issue first time in Oct 2005 and since then the issue reoccur quite randomly sometime once in a week and sometimes once in 2-3 months.

There is nothing else changed majorly with the Software, we have increase in the load due to addition of new interfaces and new polling components for the same interface maps.

To confirm that it is due to Memory leak we are planning to measure the memory consumed and correlate it with occurance of this problem,can you please inform if this can be related to Memory consumed by launcher process only or due other processes of Eventserver.

If we need to measure memory consumend by Launcher process on Unix( HP) box , do you have any commands which would be handy to measure the memory consumption.

Thank in advance,

Regards,
Sneha

 

[janhes]

We found it was the Launcher proces that caused the problem. Under windows we saw the memory usage exceeding the VM memory.

 

[OllyC]

>do you have any commands which would be handy to measure the memory consumption.
call a script like the following every 30 mins (using your monitoring tool or a cron job)

PID=`cat /mercator1/.launcherPID`
ps l $PID

(the name of the PID file might be slightly different in 671)

the memory usage is in the SZ and RSS columns, the script works for AIX, should be similar on HP...