DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
sensiva



Group memberships:
Premium Members

Joined: 22 Aug 2017
Posts: 18

Points: 344

Post Posted: Thu Nov 07, 2019 2:46 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hello All,

I am experiencing the error "The record is too big to fit in a block" in one of my job which uses MQ connector to read messages from a queue.

Scenario

    The MQ queue has around 15000 messages with the size of each message is some kb. Definelty less than a 0.5MB

    The Datastage connects to the MQ in client mode and reads the messages

    The message quantity is set to -1 so as to read all messages from the queue


I would like to better understand how the datastage works for streaming events as its says records too big to fit in a block. How datastage split MQ messages to block , how does this mapping happen internally.

I understand that the mq connector would read message by messgae and not line by line, in this case, does the connector waits until it reads all the messages in the queue before propagating it to the next stage ? or does it propagate the messages to next stage as it reads ?

Because, our transport block by default was set to 128kb and Datastage was saying it required around 750kb initially. Since it was below the recommened level of 1MB, I increased it to 900kb, but during the next run, the job again failed and said it needs 1.2MB, Though its a little above the recommended level, I could increase it to 1.2MB, but would like to understand if 1.2MB would be ok, or would it need more later ..

During functional and unit testing, we were testing with less than 10 messages in the queue, and we did not face any issues at those time.

Any pointers would be of great help.

Thanks
Sen

_________________
sen
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours