Web Service Client stage - issue with xpath definition

Dedicated to DataStage and DataStage TX editions featuring IBM<sup>®</sup> Service-Oriented Architectures.

Moderators: chulett, rschirm

Post Reply
hobocamp
Premium Member
Premium Member
Posts: 98
Joined: Thu Aug 31, 2006 10:04 am

Web Service Client stage - issue with xpath definition

Post by hobocamp »

I don't see anything that I can directly relate to this problem I'm encountering when using a Web Service Client stage - so here goes -

I'm trying to call a web service from a vendor, who supplied the wsdl definitions for the various services we need to call.

The wsdl for this particular service is only two nodes - one gets loaded with an xml payload, and the other with a string.

After I've imported the wsdl, the xpath definition for the second node type is text(). But the first node has a type of "*". Of course if you run the job with that value, it results in an "invalid token" error.

The only thing I've been able to get to work is to set the value to text(), like the second node. However, with that, DataStage interprets the xml payload with ;gt and ;lt, instead of left and right brackets. The end result is that the web service call fails.

I know that this is the cause of the failure, because if I run a trace on the job, capture the payload, convert the gt and lt values to left and right brackets, and run it in SoapUI, the call works perfectly.

(On the other hand, if I capture the data as is, and run it in SoapUI, I get the same error that I get in DataStage, which is "Object reference not set to an instance of an object".)

Has anyone run into something similar when call web services?

Thanks!
Tom
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

I guess one question for starters is.....does it ever work in a soap testing tool?

...on the & escape characters, it's hard to say what is going on...but "maybe" the xml data element will help you...this tells DataStage to ignore whatever xml character symbols may be in that string and to just leave it "as is". Not sure if that has anything to do with your problem, but it is something to try. Set it on the column level...

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
hobocamp
Premium Member
Premium Member
Posts: 98
Joined: Thu Aug 31, 2006 10:04 am

Post by hobocamp »

Hi Ernie - Thanks for your reply.

To answer your question - yep, it works perfectly in SoapUI, as long as I manually convert the < and %gt; to brackets (<>) first. (If I paste the soap envelope as is into SoapUI, I get the same error that DataStage produces.)

I had also tried setting the Data Element value to XML at the column level, but so far that has produced the same results.
JRodriguez
Premium Member
Premium Member
Posts: 425
Joined: Sat Nov 19, 2005 9:26 am
Location: New York City
Contact:

Post by JRodriguez »

Hi Tom,
Recently I ran into a similar issue. In our case the web service payload was defined to generate the payload with a node string and that caused that the returned XML payload was xml escaped ( < --&lt and > --&gt). This depend entirely on the web service provider...

We ended up downloading the payload to disk and performing a global replace to fix the XML tags...and continue processing the XML

It looks like you are facing a similar web service weakly typed defined with anode string which always by definition will encode the XML payload

Good luck!!
Julio Rodriguez
ETL Developer by choice

"Sure we have lots of reasons for being rude - But no excuses
hobocamp
Premium Member
Premium Member
Posts: 98
Joined: Thu Aug 31, 2006 10:04 am

Post by hobocamp »

Hi Julio -

Thanks for your response. What a frustrating issue this has been.

Would you mind giving me a few more details, such as how you went about determining that it was caused by the service, and how your workaround worked?

Thanks so much.

Tom
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

You can provide the input payload yourself...use the WSTransformer Stage instead of the WS Client Stage, and have a dummy Transformer or RowGen in front of it (or dummy read from a file/database...whatever is most convenient).....

Put "the payload that works" into a Derivation for a longvarchar column (only column on the link that enters the WSTransformer). Then...in the Input link "message" tab, use the pull-down for "User defined message". Select your single column.

Now do the same thing on the output side. Declare one large column, longvarchar, and specify it in the user defined message pulldown.

You are in control. The communications via HTTP is still done by the stage and established when you imported the wsdl and selected the operation, but the payload is being constructed by you.

Get that working perfectly....

THEN.....you can work on upstream to build the correct payload, using a combination of xmlOutput Stage and Transformer to clean anything up if necessary (hint: the _IN Table definition that gets created when you import the WSDL is the one that you use on the input link side of the xmlOutput Stage.

One question... You noted above "If I paste the soap envelope as is into SoapUI" .....where did you get this "soap envelope" if not from SOAPui?

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
hobocamp
Premium Member
Premium Member
Posts: 98
Joined: Thu Aug 31, 2006 10:04 am

Post by hobocamp »

Hi Ernie -

By "payload that works", I was referring to the soa envelope I copied from the Trace log of the DS job. I copied that to notepad, converted all of the escape characters back to brackets, and posted the entire thing into SoapUI. The service worked correctly in that case. Make sense?

Tom
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Either way, the solution I outlined above will do the trick. Manage the payload entirely on your own using the user-defined message methodology......

...and then work separately on ways to build that payload on the way in or to parse it on the way out, with or without xml stages, depending on the payload complexity.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
hobocamp
Premium Member
Premium Member
Posts: 98
Joined: Thu Aug 31, 2006 10:04 am

Post by hobocamp »

Thanks Ernie.

Couple of questions just to clarify -

-When you mentioned adding the 'working' payload to a derivation (inside a dummy transformer I'm assuming), do you mean the entire soa envelope content? I'm assuming yes.

-Also, you mentioned the input and xml output stage. You also mentioned having an output from that stage. What is the target for that output?

Thanks again.

Tom
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

I haven't done it in awhile, but if memory serves correctly, you just need the whole <body>. A decade or so ago I asked engineering to give us a default Table definition for this purpose...if you were ever curious why there is a "BuiltIn" Table Def called SOAPBody...well...that's why.

The trick of copying the whole thing that works into a Derivation is just to "test it". Meaning --- give a perfectly working body into the Stage and then let the Stage do the HTTP work.

If that works, then you can evaluate..."how do I build this soap payload manually"? ...that's where eventually you would replace the upstream Transformer with an xmlOutput Stage (or the xml Stage if you have been using it) or whatever else you need to "build" that exact soap content....

...in the end, you might have a Job that contains ....various source stages.....an xmlOutput Stage....a Transformer (for final adjustments)...the WSTransformer...another Transformer....an xmlInputStage....other downstream stages...

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Post Reply