DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
perspicax



Group memberships:
Premium Members

Joined: 07 Dec 2017
Posts: 14
Location: USA
Points: 119

Post Posted: Mon Feb 12, 2018 3:39 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
New to DS parallel edition. I am trying to build a DS job for a fact table which has references to 35 dimensions (via Surrogate key).

What is the best way to populate the Surrogate keys. I do not want to form a gaint SQL with left outer join to reference tables from driving table. Rather I would like to use the lookup stages. I added 4 lookup stages serially, and I see that the data being pipe-lined nicely. But job construction is taking a long time.

I am also looking to re-use the lookup in few other jobs involving fact tables because the tables have almost same number of dimensional references.

Instead of using one lookup per reference table, is it a better idea to use one lookup stage with all the reference tables?

How is this normally done in DS?
bhargav_dd



Group memberships:
Premium Members

Joined: 30 Jun 2009
Posts: 57

Points: 589

Post Posted: Tue Feb 13, 2018 8:54 pm Reply with quote    Back to top    

What is Size of your reference table because lookup consumes more memory if reference tables are of more size as it traverses entire record to perform operation
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours