help with message queues

[huskers]

Hi,

I have a process(P1) which reads data from socket and puts it on a message queue. I have another process(P2) which reads from messsage queue and loads it into database. The rate at which P1 puts it on message queue is much higher than the rate at which P2 processes the data. I get error messages saying that P1 is unable to put messages on the queue as it is full. Is there any way that i could increase the efficieny like run two instances of P2 processes so that both of them simultaneouslt read from the message queue. When i did that i observed that only one of them pings to the message queue and the other one is getting locked out. Is there any parameter that i need to set when creating the message queue so that two or more processes can read it. Any suggestions are welcom.

Thanks in advance

 

[Salem]

My guess is the hold-up is the storing in the database. Adding another database writer process will only make things worse, since you'll now have two processes contending for write access to the database.

Is P2 opening and closing the database for each message? Could it buffer messages internally then write many messages for each open/close cycle?

Could P1 send a message back across the network to implement some form of flow control?

 

[huskers]

P2 reads the messages from the queue at a particular instance opens database, updates/inserts records and closes the database.

P1 cannot send a message back across the network. Because realtime data is being sent by many processes to P1 and all P1 can do is read from a socket and put it on the queue.

 

[marsd]

huskers,
I'm with you..I think multiple db updating processes
is the answer. Without sample code it's a real pain to
try to create a similar situation to prove it though.

 

[palbano]

>> Is there any parameter that i need to set when creating
>> the message queue so that two or more processes can read it.

That information would be in the API for the Message Queue Software. What product are you using? If MQ Series there is a forum for it here at Tek-Tips.

Will two processes read and insert faster than one? Perhaps on a multiprocessor box but probably not to any noticeable degree on a single processor. Salem’s point about the database contention effecting the operations of separate processes writing to the same database and tables should not be dismissed without disproving it either.

>> P2 reads the messages from the queue at a particular
>> instance opens database, updates/inserts records and
>> closes the database.

Hopefully that does not mean the process is constantly opening and closing connections to the database. If it is doing that you can make P2 much faster by holding a single connection for the lifetime of the process.

Can you throttle P1 without losing events? If so that would certainly be something that should be done. What platform are you running in? Some inter-process synchronization mechanisms are available, depending on your platform, such that P2 could signal P1 when the Queue size is below a threashold.


-pete

 

[huskers]

Thanks for the responses. I am running it on a unix box. The message queue is created as follows:
msgget(MYQ, IPC_CREAT | 0660). P1 process reads from the sockets and puts it on the MYQ message queue using
msgsnd().

The problem is P1 process gets realtime data from lot of machines so it is not a good idea to make P2 tell P1 that message queue is full and it should stop putting messages on the queues.

I am thinking of something like when P2 reads all the messages from the queue it should release the message queue and start doing its database insertions/uodates so that if we start other process P3 then it can read the data that is there on the message queue release it and do its database insertions/updates.
OR
I have to think of creating two message queues MYQ1 and MYQ2 so that P1 process can put the data on MYQ2 once MYQ1 is full. Then i will have two processes P2 reading from MYQ1 and P3 reading from MYQ2. I know this does not solve the problem totally.

Let me know what you guys think.

Thanks in advance