DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
sandhya.budhi



Group memberships:
Premium Members

Joined: 15 Nov 2017
Posts: 18

Points: 348

Post Posted: Fri Jan 05, 2018 4:11 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
I have a similar issue to this post when I am trying to make a REST call with 7k records. The data don't have any key and I have to process in the same order received from the source.

The REST call is failing with the error

CDIER0410E: Error in step=REST, cause=com.ibm.e2.provider.ws.exceptions.RESTWSRequestException: CDIER0905E: The REST step found a large input data item whose size 23415819 exceeds the supported size limit 10485760

I tried the options provided in the linked post but it is not working. Increased the heap size and still getting the same error.

I am trying to use the option Split data into batches in the Heirarchical Stage. But I am not having any key data so not sure how to proceed with this.

Please suggest how to proceed with this

_________________
Thanks,
Sandhya
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42576
Location: Denver, CO
Points: 219102

Post Posted: Fri Jan 05, 2018 9:57 pm Reply with quote    Back to top    

Moderator: split you out to your own topic and linked back to the original.

_________________
-craig

I know I don't say this enough, but I like when you talk to me. It's much better than when nobody talks to me. Or when people that I don't like will not stop talking to me.
Rate this response:  
Not yet rated
cdp



Group memberships:
Premium Members

Joined: 15 Dec 2009
Posts: 113
Location: New Zealand
Points: 1632

Post Posted: Tue Jan 23, 2018 5:35 pm Reply with quote    Back to top    

Well...
You may have reached the limit of the Hierarchical Stage my friend:

https://www.ibm.com/support/knowledgecenter/en/SSZJPZ_11.3.0/com.ibm.swg.im.iis.ds.stages.xml.msg.doc/topics/CDIER0919E.html

You can raise a request for enhancement, if we are 2 complaining, maybe it will be added faster Laughing

As a workaround though, you can limit the amount of records you are retrieving for each REST call:
https://blogs.sap.com/2013/03/20/using-odatas-top-skip-and-count/
And then loop to get the rest of the records (usually the REST response will tell you at which record you stopped and hence, from where you need to loop to get the rest of the records)

Also, just in case, try running your job that does the REST call by adding those 2 parameters in your job properties:

Quote:
$APT_DEFAULT_TRANSPORT_BLOCK_SIZE = 999999999999999
APT_MAX_DELIMITED_READ_SIZE = 99999999999


Set them to the max allowable value (DataStage will let you know what it is)

$APT_DEFAULT_TRANSPORT_BLOCK_SIZE... you'll find it in the Environment variables

But you won't find "APT_MAX_DELIMITED_READ_SIZE" anywhere... Just enter it manually.
N.B. Don't add a $ sign for this one!


Quote:
APT_MAX_DELIMITED_READ_SIZE: Environment variable to specify the upper bound for the number of bytes InfoSphere® DataStage® looks ahead to the the next delimiter


Good luck Wink
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours