Parallel Processing of Rest Object in Rest Call

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
atul9806
Participant
Posts: 96
Joined: Tue Mar 06, 2012 6:12 am
Location: Pune
Contact:

Parallel Processing of Rest Object in Rest Call

Post by atul9806 »

Hi Gurus
There is a requirement of a REST call which has the Request as below -

Code: Select all

{
    "key1": [   "value1     ],
    "idSet": [                { 
               "id": [
        { "idType": "com.server1"
          "idValue": "451902123"
        }  ],
               "id": [
        { "idType": "com.server2"
          "idValue": "4519021234"
        }  ]
               }              
    ]
  }
Here "idSet" array object can have thousands of element "id", Can we process them parallel in Hierarchal Stage while making REST call.
~Atul Singh
<a href=http://www.datagenx.net>DataGenX</a> | <a href=https://www.linkedin.com/in/atulsinghds>LinkedIn</a>
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Depends a bit on what you mean by "process them in parallel". The REST call itself is a blocked call, so once you finish building your payload you will then issue a POST, or whatever and it will send it.....but if you are transforming or creating those IDs somewhere else, there are lots of strategies for what you might do with them beforehand, in parallel or otherwise, inside of, or upstream from, the Hierarchical Stage.

What is the goal? Do you want to process them in independent or "grouped" REST calls (with a subset of the array in each call?)?

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Post Reply