Lookup recommendation

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
perspicax
Premium Member
Premium Member
Posts: 26
Joined: Thu Dec 07, 2017 3:11 pm
Location: USA

Lookup recommendation

Post by perspicax »

New to DS parallel edition. I am trying to build a DS job for a fact table which has references to 35 dimensions (via Surrogate key).

What is the best way to populate the Surrogate keys. I do not want to form a gaint SQL with left outer join to reference tables from driving table. Rather I would like to use the lookup stages. I added 4 lookup stages serially, and I see that the data being pipe-lined nicely. But job construction is taking a long time.

I am also looking to re-use the lookup in few other jobs involving fact tables because the tables have almost same number of dimensional references.

Instead of using one lookup per reference table, is it a better idea to use one lookup stage with all the reference tables?

How is this normally done in DS?
bhargav_dd
Premium Member
Premium Member
Posts: 57
Joined: Tue Jun 30, 2009 9:38 am

Post by bhargav_dd »

What is Size of your reference table because lookup consumes more memory if reference tables are of more size as it traverses entire record to perform operation
Post Reply