DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message

Group memberships:
Premium Members

Joined: 22 Aug 2017
Posts: 21

Points: 374

Post Posted: Thu Nov 07, 2019 2:46 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hello All,

I am experiencing the error "The record is too big to fit in a block" in one of my job which uses MQ connector to read messages from a queue.


    The MQ queue has around 15000 messages with the size of each message is some kb. Definelty less than a 0.5MB

    The Datastage connects to the MQ in client mode and reads the messages

    The message quantity is set to -1 so as to read all messages from the queue

I would like to better understand how the datastage works for streaming events as its says records too big to fit in a block. How datastage split MQ messages to block , how does this mapping happen internally.

I understand that the mq connector would read message by messgae and not line by line, in this case, does the connector waits until it reads all the messages in the queue before propagating it to the next stage ? or does it propagate the messages to next stage as it reads ?

Because, our transport block by default was set to 128kb and Datastage was saying it required around 750kb initially. Since it was below the recommened level of 1MB, I increased it to 900kb, but during the next run, the job again failed and said it needs 1.2MB, Though its a little above the recommended level, I could increase it to 1.2MB, but would like to understand if 1.2MB would be ok, or would it need more later ..

During functional and unit testing, we were testing with less than 10 messages in the queue, and we did not face any issues at those time.

Any pointers would be of great help.



Premium Poster

Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3833

Points: 30939

Post Posted: Wed Dec 04, 2019 9:10 am Reply with quote    Back to top    

It's been a really long time (probably pre-11.3, but this is a feature that I would expect to be quite stable) since I played with these settings (and I've forgotten their exact names, but it's the "a ...

Ernie Ostic

Open IGC is Here!
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours