Very slow queue processing 1

[ChrissyMG]

Hello All,

maybe you can help me out...

We have MQSeries 5.0 installed on a W2K-Server. We receive messages in triggered queues, we trigger a little C-Program that moves the messages to files.
Since a couple of weeks the processing of the messages seems to be very slow - today it needed more than 20 minutes to process 40 messages.
The server has two processors and 512 MB RAM, so the server itself seems to be ok.
I found one error message that turned up 3 times since Feb, 2nd: AMQ9500 no repository storage.
I don't know what it means.

Any hint is highly appreciated !
Thanks,
Chrissy

 

[mqonnet]

Could be possible that you are using all of the memory, if the messages are Non persistent, or you could be using all of the hard disk space if they are persistent.

Whats the size of the messages and what type they are, persistent or non-persistent.



Cheers
KK

 

[ChrissyMG]

Hi mqonnet,

we have around 50 local queues which are defined as non-persistent and 50 remote queues which are persistent.

The messages are rather small, only a few KByte.

Do the queues block memory even if there are no messages in the queues ? We don't have so much trafic but if there are messages they are extremly slow processed.

Thanks,
Chrissy

 

[mqonnet]

MQ would only use memory or disk space only if there are messages on queues. Otherwise it Wont.

If this is a test system, i would suggest you to run amqsbcg/amqsget sample apps to read/retrieve the messages and see if it works fast enough. Unless there is a problem, any number of small message should be read in a "blink of an eye". If its taking 20 minutes to read 40 messages of a few KB, then there sure seems to be some problem.

I would suspect if MQ is the culprit, it is trying to write FFST's. Check in the root install\error directory to see if any *FDC files got created. If MQ is busy cutting FFSTs, then it sure wouldnt process api calls as quickly.



Cheers
KK

 

[ChrissyMG]

Hi again,
it is a production and test server at the same time - by queue names we distinguish whether the messages come from production or not. I know it is not the best solution but better for our budget;-)
I did a test with a 20 Byte message and the program amqsget - it took unbelievable 15 seconds....
I found some *FDC-files, but not a whole lot. I don't know what FFST's are (I am normally an SAP analyst, interfaces and MQSeries are a small part of my work only), can I switch the writing of FDC-Files off ?
The other thing I was thinking about is to upgrade to version 5.2.1 which is dedicated for Win2000 whereas 5.0 seems to be more for WinNT.
Thanks a lot for your hints !
Chrissy

 

[mqonnet]

Well, FFSTs are a way to report errors to the end user and they cant be turned off. They would come out whenever MQ encounters a problem. So, what you should do is try and find out why you are getting them at the first place.

And by the way, 5.2.1 which you plan to upgrade to is also old and 5.3 is the latest. And i believe all other versions are already out of service or going out of service soon. So, it is advisible to upgrade to 5.3 instead.

Also try and keep your installation updated with the latest CSDs/fixes for that version of MQ you have. These are freely available from the IBM support website.



Cheers
KK

 

[stevexff]

The only two times I've seen MQ go slowly (admittedly, not that slowly) were:

When the license key wasn't set correctly, and it was only using one processor on a 6-way box. IBM denied that this would have any effect, but the processor statistics showed 1 engine at 100% and the others almost idle. When we set the licenses correctly, and restarted MQ, the problem went away.

The other time was with persistent queues (on z/OS, so the implementation might be different). When writing large numbers of messages to the queue in the same UOW, the time to write each message started to increase exponentially as the numbers increased. By keeping the numbers of messages to about 200 or less in each UOW, it ran really quickly.

As some of your queues are persistent, do you have a long running process that handles them? If so, do you commit after each message?

 

[mqonnet]

Stevexff, a few things on this problem that we have been discussing and what Chriss has been saying.

"today it needed more than 20 minutes to process 40 messages.
The messages are rather small, only a few KByte
"

So, your theory may not really apply to this current context. In any case, even if you had 100's of messages within one UOW, i dont think there would be and should be *any* overhead to MQ processing other api calls, viz, puts/gets etc. Because message persistance or UOW has nothing to do with MQ processing requests. Its the O/s that may be at a threshold. But in that case too, you wouldnt normally see a performance hit, only that further calls to put messages might be rejected saying no more disk space available, if thats the case. As for UOW, there are understandably system limits for the resources used. And this differs from platform to platform. Even if we are near the maximum or threshold of number of messages within UOW, it *wont* harm MQ processing any requests, because it has nothing to do with the UOW.



Cheers
KK

 

[stevexff]

Curiously, my colleague has today been experiencing MQ slowdowns. I will post anything we find as a cause...

 

[ChrissyMG]

@mqonnet and stevexff

Thanks very much to the two of you for your hints - we are checking now the possibility to install 5.3 - the big problem is really our organisation, MQSeries has been introduced at our site by consultants and except me there is nobody who knows whatsoever about it. I have to find out if we have such sort of a license that we get the 5.3 package by IBM as an upgrade or not.
In the meantime I created a second trigger monitor for sensitive interfaces, hoping that they at least don't get blocked anymore by other messages which are not so important.
Another remark I got from a colleague in another division was to put the active logs onto another device than the local, persistent queues - this would help the perfomance.. But: as far as I see it I would have to reinstall the queue manager... which I really don't fancy seeing that there is no support here if I break it...
What do you think about that ?
Thanks again,
Chris

 

[ChrissyMG]

Addition:

By the way, I think there is a commit after each message - we trigger each message single and the messages are not sent in packets or so.

 

[kevinf2349]

<quote> ... we trigger each message single and the messages are not sent in packets or so. </quote>

You have a trigger type of EVERY? This may seem a strange question but...why? What does the trigger do? (e.g Start a channel, start a process).

I always try to avoid a trigger type of EVERY as I would rather have the triggered application account for multiple messages arriving on the queue then have multiple executions of the same process and the overhead of the extra trigger messages being generated.

 

[ChrissyMG]

@kevinf2349
To be honest - I don't know. We were told to have the settings like this... The trigger starts a c program which reads the message from the queue and writes it into a text file. That's all, nothing more special than that.
Maybe I can try out if trigger type First helps the performance.
Thanks for the hint !
Chris