DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
pbttbis



Group memberships:
Premium Members

Joined: 11 Dec 2014
Posts: 35
Location: South Africa
Points: 515

Post Posted: Wed Jan 27, 2016 10:07 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hi,

We are having an issue in the REST step (Hierarchy Stage) of an ISD job
where the first REST call is always successful, followed by the next
call failing, followed by the next call successful, followed by the
next failing and so on. It seems the failure seems to "reset" whatever
the issue is so that the next call is successful.

We have confirmed that the every other failure is not being received at all by the system being called in the REST step.

Added TRACE logging to the hierarchical stage and the first lines that differs in the XMLStage_REST_0.log when comparing a success and failure is:

2016-01-26 18:40:02,959 Debug [REST] [] # of XML Parser transitions in document: 0
2016-01-26 18:40:02,959 Debug [REST] [] Number of XLXP Events consumed in document: 0

Anyone experienced something like this or have some ideas?

Have logged a PMR with IBM

Thanks,

Shaun

_________________
PBT TBIS Consultant
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3773

Points: 30298

Post Posted: Wed Jan 27, 2016 3:28 pm Reply with quote    Back to top    

Hard to say....can you share with us the topology of the overall ISD Job? Is there a possibility that your Hierarchical Stage in the Job has more than one input link? Are there any Stages in the ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
pbttbis



Group memberships:
Premium Members

Joined: 11 Dec 2014
Posts: 35
Location: South Africa
Points: 515

Post Posted: Thu Jan 28, 2016 3:46 am Reply with quote    Back to top    

ISD_INPUT
-> Transformer_1 (had split to ODBC load but currently removed for testing)
-> CopyStage_1
-> CopyStage_2
-> Transformer_2
-> Tranfomer_3 (in place to work around Hierachial Stage micro seconds truncation bug)
-> Hierarchical Stage (has the REST Step)
-> Transformer_3 (in place to work around Hierachial Stage micro seconds truncation bug)
-> CopyStage_3
-> Transformer_4 (had split to ODBC load but currently removed for testing)
-> Funnel (Inputs: Tranformer_2 and Transformer_4)
-> CopyStage_4
ISD_OUTPUT

Input Message Flow:

apache httpd -> Datastage -> apache tomcat loadbalancer -> apache tomcat worker

With more testing we have found that submitting requests to the DataStage URL via the WizTools RESTClient that we get success every time.

We have confirmed that for the failures that the connection is not even received by the loadbalancer.

_________________
PBT TBIS Consultant
Rate this response:  
Not yet rated
pbttbis



Group memberships:
Premium Members

Joined: 11 Dec 2014
Posts: 35
Location: South Africa
Points: 515

Post Posted: Thu Jan 28, 2016 9:25 am Reply with quote    Back to top    

Found this link that seems to be what I am experiencing:

http://www-01.ibm.com/support/knowledgecenter/SSZJPZ_8.7.0/com.ibm.swg.im.iis.infoservdir.user.doc/topics/c_design_constraints_rt_ref_match.html

_________________
PBT TBIS Consultant
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3773

Points: 30298

Post Posted: Thu Jan 28, 2016 2:55 pm Reply with quote    Back to top    

...well, that's where I was going with my questions above, but it doesn't sound like you have that topology...... the topology in that link that is problematic, and which I was hinting at, is one that ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
pbttbis



Group memberships:
Premium Members

Joined: 11 Dec 2014
Posts: 35
Location: South Africa
Points: 515

Post Posted: Fri Jan 29, 2016 7:24 am Reply with quote    Back to top    

So the REST step performing the POST would not be considered a reference lookup and create a "real time quandry" situation?

Yeah I am working with the apache admin trying to figure out how the connection config differs between using the test tool as opposed to the actual system.

Is there anyway I can

1) check that the datastage connection setup in the REST step is in fact closed after a response is recieved
2) from the datastage server check that the connection received from the apache server is closed after the ISD_Output responds

_________________
PBT TBIS Consultant
Rate this response:  
Not yet rated
pbttbis



Group memberships:
Premium Members

Joined: 11 Dec 2014
Posts: 35
Location: South Africa
Points: 515

Post Posted: Mon Feb 01, 2016 4:55 am Reply with quote    Back to top    

Some more investigating with wireshark and tcpdumps we see the follows:

tomcat (8080) sends a fin, ack, datastage(51759) sends an ack closing the TCP connection

Datastage does not send a FIN along with its ACK, so when Datastage sends another REST request from port 51759 which tomcat knows is closed, therefore sends two RST reset packets in response.

Is there anyway in DataStage I can call a connection.close() or some setting to force a connection closed after it has been made in the REST step?

_________________
PBT TBIS Consultant
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3773

Points: 30298

Post Posted: Mon Feb 01, 2016 7:46 am Reply with quote    Back to top    

Not that I've ever seen, but anything is possible. Question --- does the same set of REST calls, using the Hierarchical Stage, work if you have a regular batch Job, with a RowGen or other type o ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
pbttbis



Group memberships:
Premium Members

Joined: 11 Dec 2014
Posts: 35
Location: South Africa
Points: 515

Post Posted: Mon Feb 01, 2016 8:52 am Reply with quote    Back to top    

Yeah as a batch job reading the URL encoded data I can submit the request successfully over and over again.

_________________
PBT TBIS Consultant
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3773

Points: 30298

Post Posted: Mon Feb 01, 2016 2:48 pm Reply with quote    Back to top    

It would be interesting, if you have the time, to put in an end-of-wave Operator upstream in your batch Job.... set a counter and just have the end of wave take action for every row. It might die t ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3773

Points: 30298

Post Posted: Mon Feb 01, 2016 2:49 pm Reply with quote    Back to top    

It would be interesting, if you have the time, to put in an end-of-wave Operator upstream in your batch Job.... set a counter and just have the end of wave take action for every row. It might die t ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
pbttbis



Group memberships:
Premium Members

Joined: 11 Dec 2014
Posts: 35
Location: South Africa
Points: 515

Post Posted: Tue Feb 02, 2016 9:22 am Reply with quote    Back to top    

Okay we have changed the plumbing a bit and the architecture now is:

apache httpd -> Datastage -> HTTP Forwarder -> apache tomcat loadbalancer -> apache tomcat worker

Seems in this configuration Apache is now handling the keep alives and re-opening of connections to tomcat. Do not understand it 100%, but its wrorking and the client is happy with the architecture.

Going to mark this thread as resolved. Thanks for the help Ernie.

_________________
PBT TBIS Consultant
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours