Segmentation for large messages

[baileyonyx]

We are running MQSeries for Web Applications / to & from the As400.

Our messages from the AS400 is too large for the buffer (32000 max) and when it gets to the web in truncates.

Can anyone give me advise on how to handle large messages?

Thanks

 

[smeridew]

The maximum message size for AS400 MQSeries V5.1 is 4MB. You should have no problem retreiving these messages - increase you buffer size. If this is not possible (can't imagine why) then you'll have to group the messages into smaller segments. Check the documentation for message groups. Here's the snippet:

This is supported on MQSeries Version 5 products and MQSeries for AS/400 only.

Messages can be too large for the application, queue, or queue manager. MQSeries provides three ways of dealing with large messages:


Increase the queue and queue manager MaxMsgLength attributes (not AS/400).

Use segmented messages. (Messages can be segmented by either the application or the queue manager.)

Use reference messages.
Each of these approaches is described in the remainder of this section.


Increasing the maximum message length

The MaxMsgLength queue manager attribute defines the maximum length of a message that can be handled by a queue manager. Similarly, the MaxMsgLength queue attribute is the maximum length of a message that can be handled by a queue. The default maximum message length supported depends on the environment you are working in, as shown under "Message properties and protocols".

If you are handling large messages, you can alter these attributes independently (not AS/400). The attribute value can be set between 32768 bytes and 100 MB.

After changing one or both of the MaxMsgLength attributes, restart your applications and channels to ensure that the changes take effect. When these changes are made, the message length must be less than or equal to both the queue and the queue manager MaxMsgLength attributes. However, existing messages may be longer than either attribute.

If the message is too big for the queue, MQRC_MSG_TOO_BIG_FOR_Q is returned. Similarly, if the message is too big for the queue manager, MQRC_MSG_TOO_BIG_FOR_Q_MGR is returned. However, these return codes can be avoided by segmenting your messages (see "Message segmentation" for full details of this).

This method of handling large messages is easy and convenient. However, consider the following factors before using it:


Uniformity among queue managers is reduced. The maximum size of message data is determined by the MaxMsgLength for each queue (including transmission queues) on which the message will be put. This value is often defaulted to the queue manager's MaxMsgLength, especially for transmission queues. This makes it difficult to predict whether a message is too large when it is to travel to a remote queue manager.

Usage of system resources is increased. For example, applications need larger buffers, and on some platforms, there may be increased usage of shared storage. Note that queue storage should be affected only if actually required for larger messages.

Channel batching is affected. A large message still counts as just one message towards the batch count but needs longer to transmit, thereby increasing response times for other messages.

Message segmentation

Increasing the maximum message length as discussed in topic "Increasing the maximum message length" has some negative implications. Also, it could still result in the message being too large for the queue or queue manager. In these cases, a message can be segmented. For information about segments, see "Message groups".

The next sections look at common uses for segmenting messages. For putting and destructively getting, it is assumed that the MQPUT or MQGET calls always operate within a unit of work. It is strongly recommended that this technique is always used, to reduce the possibility of incomplete groups being present in the network. Single-phase commit by the queue manager is assumed, but of course other coordination techniques are equally valid.

Also, in the getting applications, it is assumed that if multiple servers are processing the same queue, each server executes similar code, so that one server never fails to find a message or segment that it expects to be there (because it had specified MQGMO_ALL_MSGS_AVAILABLE or MQGMO_ALL_SEGMENTS_AVAILABLE earlier).


Segmentation and reassembly by queue manager
This is the simplest scenario, in which one application puts a message to be retrieved by another. The message may be large: not too large for either the putting or the getting application to handle in a single buffer, but possibly too large for the queue manager or a queue on which the message is to be put.

The only changes necessary for these applications are for the putting application to authorize the queue manager to perform segmentation if necessary,

PMO.Options = (existing options)
MQPUT MD.MsgFlags = MQMF_SEGMENTATION_ALLOWED

and for the getting application to ask the queue manager to reassemble the message if it has been segmented:

GMO.Options = MQGMO_COMPLETE_MSG | (existing options)
MQGET

The application buffer must be large enough to contain the reassembled message (unless the MQGMO_ACCEPT_TRUNCATED_MSG option is included).

If data conversion is necessary, it may have to be done by the getting application specifying MQGMO_CONVERT. This should be straightforward because the data conversion exit is presented with the complete message. Attempting to do data conversion in a sender channel will not be successful if the message is segmented, and the format of the data is such that the data-conversion exit cannot carry out the conversion on incomplete data.


Application segmentation
This example shows how to segment a single large message

Application segmentation is used for two main reasons:


Queue-manager segmentation alone is not adequate because the message is too large to be handled in a single buffer by the applications.

Data conversion must be performed by sender channels, and the format is such that the putting application needs to stipulate where the segment boundaries are to be in order for conversion of an individual segment to be possible.
However, if data conversion is not an issue, or if the getting application always uses MQGMO_COMPLETE_MSG, queue-manager segmentation can also be allowed by specifying MQMF_SEGMENTATION_ALLOWED. In our example, the application segments the message into four segments:

PMO.Options = MQPMO_LOGICAL_ORDER | MQPMO_SYNCPOINT

MQPUT MD.MsgFlags = MQMF_SEGMENT
MQPUT MD.MsgFlags = MQMF_SEGMENT
MQPUT MD.MsgFlags = MQMF_SEGMENT
MQPUT MD.MsgFlags = MQMF_LAST_SEGMENT

MQCMIT

If you do not use MQPMO_LOGICAL_ORDER, the application must set the Offset and the length of each segment. In this case, logical state is not maintained automatically.

The getting application cannot, or chooses not to, guarantee to have a buffer that will hold any reassembled message. It must therefore be prepared to process segments individually.

For messages that are segmented, this application does not want to start processing one segment until all of the segments that constitute the logical message are present. MQGMO_ALL_SEGMENTS_AVAILABLE is therefore specified for the first segment. If you specify MQGMO_LOGICAL_ORDER and there is a current logical message, MQGMO_ALL_SEGMENTS_AVAILABLE is ignored.

Once the first segment of a logical message has been retrieved, MQGMO_LOGICAL_ORDER is used to ensure that the remaining segments of the logical message are retrieved in order.

No consideration is given to messages within different groups. If such messages do occur, they are processed in the order in which the first segment of each message appears on the queue.

GMO.Options = MQGMO_SYNCPOINT | MQGMO_LOGICAL_ORDER
| MQGMO_ALL_SEGMENTS_AVAILABLE | MQGMO_WAIT
do while ( SegmentStatus == MQSS_SEGMENT )
MQGET
/* Process each remaining segment of the logical message */
...

MQCMIT


Application segmentation of logical messages
The messages must be maintained in logical order in a group, and some or all of them may be so large that they require application segmentation.

In our example, a group of four logical messages is to be put. All but the third message are large, and require segmentation which is performed by the putting application:

PMO.Options = MQPMO_LOGICAL_ORDER | MQPMO_SYNCPOINT

MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP | MQMF_SEGMENT
MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP | MQMF_SEGMENT
MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP | MQMF_LAST_SEGMENT

MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP | MQMF_SEGMENT
MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP | MQMF_LAST_SEGMENT

MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP

MQPUT MD.MsgFlags = MQMF_LAST_MSG_IN_GROUP | MQMF_SEGMENT
MQPUT MD.MsgFlags = MQMF_LAST_MSG_IN_GROUP | MQMF_SEGMENT
MQPUT MD.MsgFlags = MQMF_LAST_MSG_IN_GROUP | MQMF_LAST_SEGMENT

MQCMIT

In the getting application, MQGMO_ALL_MSGS_AVAILABLE is specified on the first MQGET. This means that no messages or segments of a group are retrieved until the entire group is available. When the first physical message of a group has been retrieved, MQGMO_LOGICAL_ORDER is used to ensure that the segments and messages of the group are retrieved in order:

GMO.Options = MQGMO_SYNCPOINT | MQGMO_LOGICAL_ORDER
| MQGMO_ALL_MESSAGES_AVAILABLE | MQGMO_WAIT

do while ( (GroupStatus != MQGS_LAST_MSG_IN_GROUP) ||
(SegmentStatus != MQGS_LAST_SEGMENT) )
MQGET
/* Process a segment or complete logical message. Use the GroupStatus
and SegmentStatus information to see what has been returned */
...

MQCMIT

Note: If you specify MQGMO_LOGICAL_ORDER and there is a current group, MQGMO_ALL_MSGS_AVAILABLE is ignored.


Putting and getting a segmented message that spans units of work
You can put and get a segmented message that spans a unit of work in a similar way to "Putting and getting a group that spans units of work".


Reference messages

This method allows a large object to be transferred from one node to another without the need for the object to be stored on MQSeries queues at either the source or the destination nodes. This is of particular benefit where the data already exists in another form, for example, for mail applications.

To do this, you need to specify a message exit at both ends of a channel. For information on how to do this, see Chapter 35. "Channel-exit programs" in the MQSeries Intercommunication book.

MQSeries defines the format of a reference message header (MQRMH). See "MQRMH - Message reference header" in the MQSeries Application Programming Reference manual for a description of this. This is recognized by means of a defined format name and may or may not be followed by actual data.

To initiate transfer of a large object, an application can put a message consisting of a reference message header with no data following it. As this message leaves the node, the message exit retrieves the object in an appropriate way and appends it to the reference message. It then returns the message (now larger than before) to the sending Message Channel Agent for transmission to the receiving MCA.

Another message exit is configured at the receiving MCA. When this message exit sees one of these messages, it creates the object using the object data that was appended and passes on the reference message without it. The reference message can now be received by an application and this application knows that the object (or at least the portion of it represented by this reference message) has been created at this node.

The maximum amount of object data that a sending message exit can append to the reference message is limited by the negotiated maximum message length for the channel. The exit can only return a single message to the MCA for each message that it is passed, so the putting application can put several messages to cause one object to be transferred. Each message must identify the logical length and offset of the object that is to be appended to it. However, in cases where it is not possible to know the total size of the object or the maximum size allowed by the channel, the sending message exit can be designed so that the putting application just puts a single message, and the exit itself puts the next message on the transmission queue when it has appended as much data as it can to the message it has been passed.

Before using this method of dealing with large messages, consider the following:


The MCA and the message exit run under an MQSeries user ID. The message exit (and therefore, the user ID) needs to access the object to either retrieve it at the sending end or create it at the receiving end; this may only be feasible in cases where the object is universally accessible. This raises a security issue.

If the reference message with bulk data appended to it must travel through several queue managers before reaching its destination, the bulk data is present on MQSeries queues at the intervening nodes. However, no special support or exits need to be provided in these cases.

Designing your message exit is made difficult if rerouting or dead-letter queuing is allowed. In these cases, the portions of the object may arrive out of order.

When a reference message arrives at its destination, the receiving message exit creates the object. However, this is not synchronized with the MCA's unit of work, so if the batch is backed out, another reference message containing this same portion of the object will arrive in a later batch, and the message exit may attempt to recreate the same portion of the object. If the object is, for example, a series of database updates, this might be unacceptable. If so, the message exit must keep a log of which updates have been applied; this may require the use of an MQSeries queue.

Depending on the characteristics of the object type, the message exits and applications may need to cooperate in maintaining use counts, so that the object can be deleted when it is no longer needed. An instance identifier may also be required; a field is provided for this in the reference message header (see "MQRMH - Message reference header" in the MQSeries Application Programming Reference manual).

If a reference message is put as a distribution list, the object must be retrievable for each resulting distribution list or individual destination at that node. You may need to maintain use counts. Also consider the possibility that a given node may be the final node for some of the destinations in the list, but an intermediate node for others.

Bulk data is not normally converted. This is because conversion takes place before the message exit is invoked. For this reason, conversion should not be requested on the originating sender channel. If the reference message passes through an intermediate node, the bulk data is converted when sent from the intermediate node, if requested.

Reference messages cannot be segmented.

Using the MQRMH and MQMD structures
See "MQRMH - Message reference header" and "MQMD - Message descriptor" in the MQSeries Application Programming Reference manual for a description of the fields in the reference message header and the message descriptor.

In the MQMD structure, the Format field must be set to MQFMT_REF_MSG_HEADER. The MQHREF format, when requested on MQGET, is converted automatically by MQSeries along with any bulk data that follows.

Here is an example of the use of the DataLogicalOffset and DataLogicalLength fields of the MQRMH:

A putting application might put a reference message with:

No physical data
DataLogicalLength = 0 (this message represents the entire object)
DataLogicalOffset = 0.
Assuming that the object is 70,000 bytes long, the sending message exit sends the first 40,000 bytes along the channel in a reference message containing:

40,000 bytes of physical data following the MQRMH
DataLogicalLength = 40,000
DataLogicalOffset = 0 (from the start of the object).


It then places another message on the transmission queue containing:


No physical data

DataLogicalLength = 0 (to the end of the object). You could specify a value of 30,000 here.

DataLogicalOffset = 40,000 (starting from this point).
When this message exit is seen by the sending message exit, the remaining 30,000 bytes of data is appended, and the fields are set to:

30,000 bytes of physical data following the MQRMH
DataLogicalLength = 30,000
DataLogicalOffset = 40,000 (starting from this point).
The MQRMHF_LAST flag is also set.

For a description of the sample programs provided for the use of reference messages, see Chapter 31, "Sample programs (all platforms except OS/390)".


--------------------------------------------------------------------------------

© IBM Corporation 1999. All Rights Reserved