Mutiple Lookup

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
shukla_krishna
Premium Member
Premium Member
Posts: 28
Joined: Mon Apr 20, 2009 11:27 pm
Location: san antonio

Mutiple Lookup

Post by shukla_krishna »

I need to do around 200 lookup to get descriptions for every code for every rows. Doing 200 Lookup is ugly design. same is needed for multiple tables.


Input rows are around 10Mill Records. Tried Parallel routine, but its slowing down.


Can some one suggest me what need to be done.
Krishna Shukla
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Join?
-craig

"You can never have too many knives" -- Logan Nine Fingers
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Re: Mutiple Lookup

Post by ray.wurlod »

shukla_krishna wrote:Doing 200 Lookup is ugly design.
I disagree. You do what you need to do.

Tip: spread things out. The design canvas is infinitely wide and deep. Add documentation (annotations) to indicate which lookup is being done where, and supplement that with meaningful stage and link names.

Spend some time monitoring this large job; it is an example where operator combination will be too aggressive; use selective stage-based denial of combinability to manage the processing load better.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Joining to the same table 200 times won't be "pretty" either but as noted, you have to do what you have to do to get the job done. Be it source joins or lookups.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply