https for data retrieval?
https for data retrieval?
I'm at a customer site where a vendor just asked if DataStage can connect and retrieve data through an https connection (from the way it was phrased, I assume without using WSDL).
I've requested more info from the vendor and I'm wading through the manual right now, but also thought I'd throw this up for grabs in case anyone can definitively say either "No" or "look at this particular stage".
I've requested more info from the vendor and I'm wading through the manual right now, but also thought I'd throw this up for grabs in case anyone can definitively say either "No" or "look at this particular stage".
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The answer's always "yes", since the "how" question wasn't asked. At worst, you could use External Source stage plus a hand-coded application.
UniVerse (but not, alas, DataStage) has its own CallHTTP library, which would have made life very easy. Maybe you could grab that UniVerse manual and reproduce the functions as parallel routines or objects.
No doubt there are easier ways through WISD - we await Ernie's next visit for more on that.
UniVerse (but not, alas, DataStage) has its own CallHTTP library, which would have made life very easy. Maybe you could grab that UniVerse manual and reproduce the functions as parallel routines or objects.
No doubt there are easier ways through WISD - we await Ernie's next visit for more on that.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
I've seen it done via JavaPack and also with the old Perl Stage that we used to have. Should be fairly simple for anyone who has done http inside of a Java class.....
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
...of course, I forgot to ask what they wanted to get via http...I was assuming they just want a document or something at a url.... are they really asking if they can get to a SOAP based web service, or maybe a REST based web service?
The answer will be the same, unless it's SOAP that WSPack can handle...the Stages in WSPack support https....
Ernie
The answer will be the same, unless it's SOAP that WSPack can handle...the Stages in WSPack support https....
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
This is still the "Very large files through Cast Iron" problem. I had the server job with row splitter working, then Cast Iron encountered a java heap error caused by a java bug that won't be fixed until the next major release in 2010.
The Engineers from Cast Iron want to get away from using the WSDL and think it can be sent through one of their https interfaces. Now I have to see if we can receive the same. I'm trying to get more info from them now.
-Edit-
Just had a conversation with an engineer from Cast Iron. It turns out they compete with WebLogic and to a certain extent with DataStage. They provide provide prewritten interfaces to dozens of external applications - way more than DataStage. Their strength is their ability to do B-to-B and B-to-Web interfaces for these off-the-shelf apps. They don't do parallel, so they aren't that great for large transfers (where DataStage shines).
In any event, now that we understand a bit more about each product, they are going to try to keep a WSDL interface, but change the driving mechanism for what is actually transferring the files. I'll post if it works!
The Engineers from Cast Iron want to get away from using the WSDL and think it can be sent through one of their https interfaces. Now I have to see if we can receive the same. I'm trying to get more info from them now.
-Edit-
Just had a conversation with an engineer from Cast Iron. It turns out they compete with WebLogic and to a certain extent with DataStage. They provide provide prewritten interfaces to dozens of external applications - way more than DataStage. Their strength is their ability to do B-to-B and B-to-Web interfaces for these off-the-shelf apps. They don't do parallel, so they aren't that great for large transfers (where DataStage shines).
In any event, now that we understand a bit more about each product, they are going to try to keep a WSDL interface, but change the driving mechanism for what is actually transferring the files. I'll post if it works!
ok.... sounds good. I wonder if they are related to StrikeIron.
Anyway...just in case YOUR side blows up with a heap error, use your right mouse button on the WSTransformer and open it up in Grid Style....if you scroll down you will see a property for java options...there you can increase the heap size used by the WSTransformer....... using -Xms syntax (I can't remember it right now, but it's the syntax where you set the maximum meg for the jvm).
Ernie
Anyway...just in case YOUR side blows up with a heap error, use your right mouse button on the WSTransformer and open it up in Grid Style....if you scroll down you will see a property for java options...there you can increase the heap size used by the WSTransformer....... using -Xms syntax (I can't remember it right now, but it's the syntax where you set the maximum meg for the jvm).
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Grid style looks very cool.
I must say, I'm not impressed with the documentation though. It definitely assumes that you are using a WSDL to load the transformer and doesn't give a great description of the properties if you are looking to manually set or tweak them.
- but back to the original problem -
Ok - the Cast Iron Developer has changed the "guts" of the call and moved it to a new port. He has not changed service names or anything else. Though it is still using the same WSDL-style calling interface, it is using a different mechanism internally to fulfill the request.
When he uses oXygen (www.oxygenxml.com) to test the interface, all he does is point it to the new port, and the call works flawlessly.
When I switch DataStage to point to the new port I'm getting the following error messages:
Cast Iron isn't seeing any problems at all - it sees a good incoming call and thinks it is supplying the info that is requested. However, DataStage is obviously having difficulty understanding the message.
Any ideas? I've tried playing with trace options, etc. and I can't get any more / different messages to figure out why it can't understand the reply.
-EDIT-
More Info - called the service from another client and got more info on the error:
So it looks like a null is coming back that is throwing things out of kilter.
I must say, I'm not impressed with the documentation though. It definitely assumes that you are using a WSDL to load the transformer and doesn't give a great description of the properties if you are looking to manually set or tweak them.
- but back to the original problem -
Ok - the Cast Iron Developer has changed the "guts" of the call and moved it to a new port. He has not changed service names or anything else. Though it is still using the same WSDL-style calling interface, it is using a different mechanism internally to fulfill the request.
When he uses oXygen (www.oxygenxml.com) to test the interface, all he does is point it to the new port, and the call works flawlessly.
When I switch DataStage to point to the new port I'm getting the following error messages:
Code: Select all
wsxfm_SJMFTPGetService,0: Fatal Error: Fatal: 0026`:`com.ascentialsoftware.wsclient.ServiceInvocationException: Service invocation exception:
at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1602)
at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1544)
at com.ascentialsoftware.wspack.Transformer.process(Transformer.java:104)
at com.ascentialsoftware.jds.StageFactory.runStage(StageFactory.java:544)
at com.ascentialsoftware.jds.StageFactory.runTransformer(StageFactory.java:361)
Caused by: java.lang.NullPointerException
at java.io.FilterInputStream.read(FilterInputStream.java:89)
at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at org.apache.axis.encoding.DeserializationContextImpl.parse(DeserializationContextImpl.java:242)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:538)
at org.apache.axis.Message.getSOAPEnvelope(Message.java:376)
at com.ascentialsoftware.wsclient.OperationCall.handleRemoteException(OperationCall.java:366)
at com.ascentialsoftware.wsclient.OperationCall.invoke(OperationCall.java:274)
at com.ascentialsoftware.wsclient.WebService.callServiceInternal(WebService.java:1685)
at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1591)
Any ideas? I've tried playing with trace options, etc. and I can't get any more / different messages to figure out why it can't understand the reply.
-EDIT-
More Info - called the service from another client and got more info on the error:
Code: Select all
Error: SOAP fault message: <S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns2:Fault xmlns:ns2="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns3="http://www.w3.org/2003/05/soap-envelope"><faultcode>ns2:VersionMismatch</faultcode><faultstring>Couldn't create SOAP message. Expecting Envelope in namespace http://schemas.xmlsoap.org/soap/envelope/, but got null </faultstring></ns2:Fault></S:Body></S:Envelope>
Error: Exception while processing web service activity. Error message: Couldn't create SOAP message. Expecting Envelope in namespace http://schemas.xmlsoap.org/soap/envelope/, but got null . Caused by: Couldn't create SOAP message. Expecting Envelope in namespace http://schemas.xmlsoap.org/soap/envelope/, but got null.
Error: SOAP fault: <?xml version="1.0"?> <ns2:Fault xmlns:ns2="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns3="http://www.w3.org/2003/05/soap-envelope"><faultcode>ns2:VersionMismatch</faultcode><faultstring>Couldn't create SOAP message. Expecting Envelope in namespace http://schemas.xmlsoap.org/soap/envelope/, but got null </faultstring></ns2:Fault>
Error:Encountered fault of type (http://schemas.xmlsoap.org/soap/envelope/, VersionMismatch), please see rest of the log for more information
Couple of things....
a) the Grid style lets you get to the java options....but as you said, the stage relies heavily on your import of WSDL and the "Select" of a particular service once imported. A lot goes on in this Stage.
b) it appears that there is something wrong with the Server side...if another client also blew up, it would seem that the host is not meeting the contract...either not creating the namespaces as required, or doing someting incorrect with the xml structure, etc. Perhaps he needs to produce a new WSDL after his revision.
Ernie
a) the Grid style lets you get to the java options....but as you said, the stage relies heavily on your import of WSDL and the "Select" of a particular service once imported. A lot goes on in this Stage.
b) it appears that there is something wrong with the Server side...if another client also blew up, it would seem that the host is not meeting the contract...either not creating the namespaces as required, or doing someting incorrect with the xml structure, etc. Perhaps he needs to produce a new WSDL after his revision.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
It was the a namespace issue. It is cleared up, and the call is working with small files. Put in a 500mb file and it fails with this error:
Is this the java heap error you were warning me about? I'm trying -Xmx512M in the Java options for input / output. Is there any way to tell if this is occurring on the remote end or the local end?
Code: Select all
sjd_HMS_BigFile_load..wsxfm_SJMFTPGetService: java.lang.OutOfMemoryError
at java.lang.String.intern(Native Method)
at java.lang.Class.getName(Class.java:1089)
at com.ibm.oti.vm.AbstractClassLoader.getPackageName(AbstractClassLoader.java:363)
at com.ibm.oti.vm.BootstrapClassLoader.loadClass(BootstrapClassLoader.java:71)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at org.apache.axis.encoding.DeserializationContextImpl.parse(DeserializationContextImpl.java:242)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:538)
at org.apache.axis.Message.getSOAPEnvelope(Message.java:376)
at org.apache.axis.client.Call.invokeEngine(Call.java:2583)
at org.apache.axis.client.Call.invoke(Call.java:2553)
at org.apache.axis.client.Call.invoke(Call.java:1753)
at com.ascentialsoftware.wsclient.OperationCall.invoke(OperationCall.java:266)
at com.ascentialsoftware.wsclient.WebService.callServiceInternal(WebService.java:1685)
at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1591)
at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1544)
at com.ascentialsoftware.wspack.Transformer.process(Transformer.java:104)
at com.ascentialsoftware.jds.StageFactory.runStage(StageFactory.java:544)
at com.ascentialsoftware.jds.StageFactory.runTransformer(StageFactory.java:361)
Sometimes its hard to tell, although it's unlikely that you'd even see the error if it was happening at the server --- you would just receive a soap fault that something died.
Go back to the beginning. We talked about this. There are too many limits here --- the WSTransformer for one, and the link/column sizes, especially under EE, as another........No way the WSTransformer will handle things much larger than 300 to 500 Meg...it uses similar techniques as the XMLInput Stage. I know it's not an answer that will help here, but there are better ways than SOAP to send this large a quantity of data.
You can try increasing the heap size further, but you may have reached the limit.
Ernie
Go back to the beginning. We talked about this. There are too many limits here --- the WSTransformer for one, and the link/column sizes, especially under EE, as another........No way the WSTransformer will handle things much larger than 300 to 500 Meg...it uses similar techniques as the XMLInput Stage. I know it's not an answer that will help here, but there are better ways than SOAP to send this large a quantity of data.
You can try increasing the heap size further, but you may have reached the limit.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Yup - I agree, but I'm limited here by either corporate restrictions or resource restrictions. The person in charge of CastIron isn't experienced, and is having difficulty redesigning it into what it should have been in the first place - a row-by-row transfer.
I've got an alternative plan where I just send him a file name to be decrypted and stored on the CastIron server. Once that is done I can use SFTP to load the data.
Of course, that assumes that I can get a patch from IBM customer service to fix the SFTP stage (it isn't working).
FYI - I did have him call the service locally (in CastIron) and it failed - so the out of memory error is on their process, not ours.
I've got an alternative plan where I just send him a file name to be decrypted and stored on the CastIron server. Once that is done I can use SFTP to load the data.
Of course, that assumes that I can get a patch from IBM customer service to fix the SFTP stage (it isn't working).
FYI - I did have him call the service locally (in CastIron) and it failed - so the out of memory error is on their process, not ours.
Yes - PX job, FTP Enterprise stage. Using SFTP, URI starting sftp:// and Port 22. We've played around with all the options, and no matter what we do we get:
SFTP: Error occurred during InitializeFromArgs()
Informative ain't it...
We have ssh installed and configured and it is working and allows seamless connections for both ssh and manual sftp's. IBM has duplicated the issue and is forwarding it to engineering - but if you have any light you can shed please do so!
SFTP: Error occurred during InitializeFromArgs()
Informative ain't it...
We have ssh installed and configured and it is working and allows seamless connections for both ssh and manual sftp's. IBM has duplicated the issue and is forwarding it to engineering - but if you have any light you can shed please do so!