DSXchange

We are in process of designing a new ETL process here. Our requirement is to extract messages from MQ and post it to destination URL by making an HTTP post. The extract job will be scheduled and the job should pick up all the messages in the queue in one go and then do the data transformation and do the data posting. In this regard I have the below queries

1. Can we do an HTTP post from Data stage?

2. Consider the scenario where we have 10 messages to be posted using HTTP and the HTTP post job fails after sending 4th message.
a. What will happen to the remaining messages pending to be sent out?
b. What will happen to the ones that are already sent out?

3. Consider the scenario where the 4th message failed in transformation job. Can we continue with the rest of the messages discarding the one that is failed?

There is nothing built-in for HTTP post.... JavaPack is the most immediate method I would think of using to do this, if only becuase it's fairly easy to implement a java class with DS using JavaPack, and it's fairly straightforward to do HTTP work inside of a java class....

...as for error handling, nothing is assured with HTTP, so that logic would have to exist in your java class, or with your integration with DataStage, probably dependent upon the relationship between "rows" in DataStage and the "messages" you are reading from MQ and/or posting to HTTP.

You have lots of options, but all are just creative ways of using DataStage..... create audit files, browse the MQ queue instead of destructively reading it, let your java class save messages somewhere on its own, etc.

Ernie

what is the Java pack you mentioned in your reply ? Kindly help me to understand this.

Here is a link to the (easily searched for) IBM InfoSphere DataStage Java Pack documentation.