Need to Publish 20,000 rows through ISD output stage SOAP

Dedicated to DataStage and DataStage TX editions featuring IBM<sup>®</sup> Service-Oriented Architectures.

Moderators: chulett, rschirm

Post Reply
chandragsc2185
Participant
Posts: 19
Joined: Thu Mar 06, 2014 12:41 pm

Need to Publish 20,000 rows through ISD output stage SOAP

Post by chandragsc2185 »

Hi,

I have a DataStage job which reads data from DB2 connector stage and outputs through ISD output stage. I have configured this job through ISD on the information server to publish the result through SOAP over HTTP protocol.I am able test this web service when the output from the DataStage job is 20 or 25 rows but when i increase to fetch all rows then the job aborts saying time out etc. Please help me how can i publish all 20,000 records?
All your help is highly appreciated.
chandrakothapalli
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Have you confirmed that the service is designed such that it can actually consume that much information at once? You may need to feed it something more... bite sized. Multiple times.
-craig

"You can never have too many knives" -- Logan Nine Fingers
chandragsc2185
Participant
Posts: 19
Joined: Thu Mar 06, 2014 12:41 pm

Post by chandragsc2185 »

I am sorry I didn't understand your comment/question?
chandrakothapalli
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Just because you want to send all 20,000 records at once to the service doesn't mean it can handle that many / an XML that large. Check with the service owner / author first to see if that is even possible.

Otherwise you may need to loop through the rows and send them individually or in smaller (like your 20 to 25 record) groups.
-craig

"You can never have too many knives" -- Logan Nine Fingers
chandragsc2185
Participant
Posts: 19
Joined: Thu Mar 06, 2014 12:41 pm

Post by chandragsc2185 »

Hi Chulett,

this what I am doing. I have designed the datastage job as defined in my earlier post. Then I have deployed the service through Information services director. I am testing that service through SOAP UI. So, its me who is the author rite and I dont know what is the limitation here. Is there a limitation on using this service on the DS, I mean In general ppl use these kind of jobs just for very small amount of data. In case if I have to publish all 20000 rows what is work around design please.
chandrakothapalli
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Sorry, for some reason I thought you had a job that pulled rows from DB2 and sent the XML to another service. Going far too fast today. :(

Show us the abort / timeout error messages. You need to determine what part of your design is the culprit. It's been far too long but I believe as a test you can remove the ISD output stage so that it simply writes to disk instead and see if you still have the issue.
-craig

"You can never have too many knives" -- Logan Nine Fingers
chandragsc2185
Participant
Posts: 19
Joined: Thu Mar 06, 2014 12:41 pm

Post by chandragsc2185 »

Please find the error messages below:

ISD_Output_2,0: PutRow Failed: dspipe_wait(19524): Writer timed out waiting for Reader to connect.
ISD_Output_2,0: The runLocally() of the operator failed.
ISD_Output_2,0: Operator terminated abnormally: runLocally() did not return APT_StatusOk.

so basically I understand it has a time out error and on the ISD output I have increased the time out to 2000 seconds, biut that didn't helped.

my job design is :

db2 stage(1000 rows,348 columns) ---> Transformer ---> ISD output.
chandrakothapalli
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

First try Craig's suggestion and prove you get complete, correct output to a Sequential File stage instead of the ISD Output stage.

You may be running into the 8.x bug where the ISD stage's buffer size setting is actually a hard limit (not the expected behavior). The default is 512 KB. Try doubling the buffer size, recompiling, and redeploying, then testing it again. Lather. Rinse. Repeat.

I returned 50 thousand rows as a test last year on version 8.7 (no fix pack) by increasing the buffer size to get past the bug. I'm sure the upper limit is much higher.

One bad thing about XML is how incredibly inefficient it is. If you look at a typical SOAP payload, you may find that 90% to 95% of the payload size is consumed by the XML opening and closing tags. That's a lot of overhead. Take that times 345 columns 20 thousand records and your web service consumer's round trip is likely to be measured minutes instead of milliseconds.
Choose a job you love, and you will never have to work a day in your life. - Confucius
chandragsc2185
Participant
Posts: 19
Joined: Thu Mar 06, 2014 12:41 pm

Post by chandragsc2185 »

Hi,

I tried the Sequential file as output and i Can get all the 345 columns and 20000 rows. Its only when the ISD stage is added i get the above mentioned error.
Actually I am running on v9.1.2 and on windows platform. I can try increasing the buffer size and will post the result. Thank you.
chandrakothapalli
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

The ISD Output stage's buffer problem was reported as fixed in 9.1 and 8.7 fix pack 1 and 8.5 fix pack 3. Details are on the IBM Support Portal under JR41343.

Have you tried reporting it to Support?
Choose a job you love, and you will never have to work a day in your life. - Confucius
chandragsc2185
Participant
Posts: 19
Joined: Thu Mar 06, 2014 12:41 pm

Post by chandragsc2185 »

I haven't reported to the IBM tech support but I have tested this job both on server and parallel and on both the jobs I get the same error:

IIS-DSEE-RTIO-00005 ISD_Output_2,0: PutRow Failed: dspipe_wait(22176): Writer timed out waiting for Reader to connect.

I have the buffer size to be 1024 and the time out to be 300.

This time i have limited to 1000 rows and 348 columns.
chandrakothapalli
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

You have more than enough info gathered to contact Support. It's probably a good idea to open a support case at this point.
Choose a job you love, and you will never have to work a day in your life. - Confucius
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Yuk. 20k rows in a SOAP based service? It can be done, but its not a best practice for any SOAP based scenario. Regular SOAP has never been a reliable transport mechanism for a "lot" of data. The major vendors (for example, review the calls available for SF.com) put a "batch" parameter inside of most of their calls for exactly this reason...you decide the size of your batch and the host sends "that" many rows. It is then predictable and manageable across any environment or scenario.

I'm not surprised it hangs. It might work in some configurations and not others.

Only other possibility is if, by chance, there is a particular row or record containing some bad data that is causing your symptoms and making it "look like" a volume issue.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Post Reply