DataStage with target on cloud

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

Neeraj,

I am a bit puzzled by your description. In it you state that DataStage is on-site at the customer, as well as the source data. If that is the case, then all data is already present on-site.

However, to answer what I believe to be the underlying question - the DataStage server will dump data to the scratch location(s) whenever the memory buffers between stages (default is 2 MB) exceeds capacity during processing. Other operations, like sorts or aggregations can also dump data to disk as required to handle large datasets.

Now if you use a file from Azure as a source or reference, like during a change capture operation or a lookup - then data from that table is definitely on the server and could be dumped to disk temporarily during buffering operations.

If the Azure data files are true targets, nothing is retrieved from them.
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

Most every client has secure data in their source systems and in their target systems and flowing through DataStage (or some ETL tool).
Choose a job you love, and you will never have to work a day in your life. - Confucius
Post Reply