Split files in DataStage

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
ksingh83
Premium Member
Premium Member
Posts: 1
Joined: Tue Aug 15, 2017 11:24 am

Split files in DataStage

Post by ksingh83 »

Hi

We have very high volume of source around 20 millions records. How can we effectively load target table?

Is the splitting of files only option? Can we use loops as another option?

Thanks
UCDI
Premium Member
Premium Member
Posts: 383
Joined: Mon Mar 21, 2016 2:00 pm

Post by UCDI »

high volume is relative ... we have loaded more than 5x that, it just takes time. You are not pushing any limits of the DS tool if your environment/server can handle it, in other words.

Did you configure the database connector load stage for a bulk load instead of insert mode?

What kind of DB is the target?
Loops work, splitting up the work can do it, there may be other system specific tricks. What do you need from it? A long running large load isnt always bad, if you don't mind waiting for it.
Post Reply