Happy New year to you all,
Any experiences you share pushing data through Layer7 secure gateway to a downstream consumer in DS9x. I appreciate any inputs.
Push Json through Layer7 gateway
Moderators: chulett, rschirm, roy
-
betterthanever
- Participant
- Posts: 152
- Joined: Tue Jan 13, 2009 8:59 am
Noticed also your other post. ; )
Level 7 ...haven't seen it come up, and this is just a guess from a quick search, but it looks like some sort of hardware based gateway/firewall/SOA processor, not unlike IBMs own DataPower.
Theoretically, one might anticipate that it can do just about anything... (DataPower, for example, can do some very sophisticated things with xml and other incoming payloads, transforming them, wrapping them, applying WS security, etc. etc. etc.)....which means that hopefully it could provide a very simplistic REST or SOAP based Web Service that DataStage could invoke, passing data on a link in either a row by row or large chunk of either JSON or XML data.
REST would eliminate the need for any WSDL...but of course will require that they put together an xsd or at least provide you with a "sample" of the payload required. Best would be a single payload for all the rows you want to send, but of course, that makes no sense if you are talking about 10000000000 rows. Then you would need some sort of a sequencing and batching mechanism (like grouping the "sends" into JSON or xml units of 100 rows apiece [for example]).
Tell the owners of the Level 7 services to keep it simple.....straightforward, row based relational xml, and initally for simpler testing and validation, NO security. Then as things get working and you are more familiar with it, start adding in additional complexity, SSL, etc.
Ernie
Level 7 ...haven't seen it come up, and this is just a guess from a quick search, but it looks like some sort of hardware based gateway/firewall/SOA processor, not unlike IBMs own DataPower.
Theoretically, one might anticipate that it can do just about anything... (DataPower, for example, can do some very sophisticated things with xml and other incoming payloads, transforming them, wrapping them, applying WS security, etc. etc. etc.)....which means that hopefully it could provide a very simplistic REST or SOAP based Web Service that DataStage could invoke, passing data on a link in either a row by row or large chunk of either JSON or XML data.
REST would eliminate the need for any WSDL...but of course will require that they put together an xsd or at least provide you with a "sample" of the payload required. Best would be a single payload for all the rows you want to send, but of course, that makes no sense if you are talking about 10000000000 rows. Then you would need some sort of a sequencing and batching mechanism (like grouping the "sends" into JSON or xml units of 100 rows apiece [for example]).
Tell the owners of the Level 7 services to keep it simple.....straightforward, row based relational xml, and initally for simpler testing and validation, NO security. Then as things get working and you are more familiar with it, start adding in additional complexity, SSL, etc.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
-
betterthanever
- Participant
- Posts: 152
- Joined: Tue Jan 13, 2009 8:59 am
coming back here to update. Now I have a little more understanding on the architecture.
yes. we need to hand shake with layer 7 through HTTP client to get a secure token first. once it is received include the token in the JSON transaction and invoke the target API URL.
any experiences on throughput of Java Integration stage to call a Java program to invoke an API. I just tested with a POC invoking a public API("http://jsonplaceholder.typicode.com/posts") with POST method. It took 28 min to post and receive response for 1000 rows. i know it depends on multiple things. Just wanted to see how good it would perform at best. I appreciate any inputs.
yes. we need to hand shake with layer 7 through HTTP client to get a secure token first. once it is received include the token in the JSON transaction and invoke the target API URL.
any experiences on throughput of Java Integration stage to call a Java program to invoke an API. I just tested with a POC invoking a public API("http://jsonplaceholder.typicode.com/posts") with POST method. It took 28 min to post and receive response for 1000 rows. i know it depends on multiple things. Just wanted to see how good it would perform at best. I appreciate any inputs.
-
betterthanever
- Participant
- Posts: 152
- Joined: Tue Jan 13, 2009 8:59 am
i just came across cURL, a command line tool, I was able to invoke the Public API right away with a simple command. Anyone used it before to invoke REST API ?
i tried this and it came back immediately. I was thinking do the handshake with layer 7 get the secure token and run a script to invoke the API?
Request:
curl -X POST -H "Content-Type:application/json" http://jsonplaceholder.typicode.com/posts -d '{"title": "foo","body": "bar","userId":1}'
Response:
{
"title": "foo",
"body": "bar",
"userId": 1,
"id": 101
}
i tried this and it came back immediately. I was thinking do the handshake with layer 7 get the secure token and run a script to invoke the API?
Request:
curl -X POST -H "Content-Type:application/json" http://jsonplaceholder.typicode.com/posts -d '{"title": "foo","body": "bar","userId":1}'
Response:
{
"title": "foo",
"body": "bar",
"userId": 1,
"id": 101
}
I've seen people use curl in a before job script and then parse the resulting xml or json in DS.....
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
-
betterthanever
- Participant
- Posts: 152
- Joined: Tue Jan 13, 2009 8:59 am
Exactly thinking in the same lines so i don't need to worry about java libraries etc...but when i tried to loop through 1000 iterations it took more than 50 min..not sure if it has to do with the public API i tried to invoke or something else..i could really use some through put numbers...the same took 28 min by invoking a java program extending processor class under java integration stage..
I'm working with a team that is building an interface for a client using DataStage / Layer7 / JSON files. We're driving the Layer7 interactions with Java programs that are triggered by DataStage jobs that have Java Integration stages. Java's the actual transport mechanism with DataStage providing the updated / re-formatted JSON files.
So far we haven't seen any performance issues, but we are transmitting entire files, not row-based communications.
So far we haven't seen any performance issues, but we are transmitting entire files, not row-based communications.
-
betterthanever
- Participant
- Posts: 152
- Joined: Tue Jan 13, 2009 8:59 am
I appreciate all of your inputs. coming back here to update.
We have setup this up. created JSONs in datastage and invoking a JAVA program/class from Java Integration stage and persist API response to a DB. we tested this in Lower Env and getting 24 Rows/sec. which is expected due to target end point being sandbox. Through layer7 we are throttled at 200rows/sec max anyway. so expecting to hit that 200 rows/sec throughput in higher env. will be happy assist if anyone has any issues.
We have setup this up. created JSONs in datastage and invoking a JAVA program/class from Java Integration stage and persist API response to a DB. we tested this in Lower Env and getting 24 Rows/sec. which is expected due to target end point being sandbox. Through layer7 we are throttled at 200rows/sec max anyway. so expecting to hit that 200 rows/sec throughput in higher env. will be happy assist if anyone has any issues.