DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
spdsquared2
Participant



Joined: 01 Jun 2015
Posts: 7
Location: Northern New Jersey
Points: 81

Post Posted: Tue Jun 12, 2018 6:12 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
I have an ISD/Rest service that does a simple db2 lookup via a datastage job (job has ISD input/output stages and lookup with sparse db2 query). The job allows caller to perform lookup on customer name, it allows user to perform wildcard searches.

When the query result is a large I only receive a partial result set back at the web browser, when I make a subsequent call (even with different query parameter) I receive the second page of results from the first query. I've been searching the web/documentation and can't find answers to my questions:
1) Is the limit configurable, if so where, is it done by #rows or total bytes returned??
2) Since it appears server is holding the subsequent results (i.e. second page of results) what's the best way to handle this... is there something sent back in client's request that let's them know more data pending to be sent?
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3773

Points: 30298

Post Posted: Tue Jun 12, 2018 8:07 am Reply with quote    Back to top    

There is no formal buffering logic built into ISD. You would have to build that intelligence/paging yourself, so that it would understand subsequent calls. ISD is effectively "stateless", and ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2789
Location: USA
Points: 21033

Post Posted: Tue Jun 12, 2018 8:33 am Reply with quote    Back to top    

The pagination described sounds like it is unwanted behavior. Perhaps it is a defect. I have not run into that myself.

I did run into large requests and results getting truncated back on version 8.7. We opened a PMR about it with Support. The workaround we used was to edit the properties of the ISD Input and Output stages. There you will find a Buffer Size setting under Options. The default size is 512 KB. We found in testing that our largest requests and responses required a size of 4096 KB. Support had described the buffer as not working as intended, and when it got full it was just truncating.

You might try disabling the ISD app, increasing those Buffer Size values, recompiling, and enabling the ISD app again. If you still see pagination then I would say to contact Support.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2789
Location: USA
Points: 21033

Post Posted: Tue Jun 12, 2018 8:42 am Reply with quote    Back to top    

Also, if you want to help protect your ISD app and the size of the load on the network, then you can add your own logic into the job design. We did this on a few jobs that were similar in design to what you have described.

You can count the number of records in the result set and limit the output to the first N records, where N is either hard-coded or passed in as an ISD Input stage column. And if you want, you can include an indicator column in the ISD Output stage to tell the consumer whether or not they reached the limit.

Generally, if your consumers are searching on something and get a large result set, you want to ask what the likelihood is that they're going to pour through all the records. Beyond some number, it may be better to suggest narrowing the search.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
spdsquared2
Participant



Joined: 01 Jun 2015
Posts: 7
Location: Northern New Jersey
Points: 81

Post Posted: Thu Jun 14, 2018 8:42 am Reply with quote    Back to top    

Sorry for the delay in responding, been nutty lately and didn't get back to this till yesterday evening.

I changed buffer size on both ISD input/output stages but it didn't seem to make a difference with the number of records returned. I'm going to limit the records returned in the DS job (like you suggested) and will probably open a ticket (also like you suggested).

Thank you for the responses, I really appreciate it!!
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours