DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message

Joined: 25 Mar 2014
Posts: 4
Location: Chennai
Points: 48

Post Posted: Fri Jul 13, 2018 4:00 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
I have datastage job which reads data from BDFS stage. The underlying HQL is currently reading data from single table (Hive Hql). All the other lookup / joins are done in Datastage job.
What would be the advantage if I perform the joins with multiple tables in Hive hql and have the data read from BDFS file stage instead of performing the joins in Datastage after reading data from single table?



Premium Poster

since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43011
Location: Denver, CO
Points: 221947

Post Posted: Fri Jul 13, 2018 8:04 am Reply with quote    Back to top    

I'm going to give some general advice here on the topic, meaning nothing specific to BDFS or Hive. The general rule of thumb is you do joins within the job flow when you need to join either disparate data sources or sources like database tables that come from separate systems. As a corollary to that, if you've got multiple tables in the same relational source, do the joins there - with a caveat. I've been at clients where the source system is the weakest link in the chain and we weren't "allowed" to do joins there for fear of impacting performance. So in those cases, we simply streamed data out as gently as we could, loaded them into our processing or target database and did the joins there if possible. Otherwise, they were done in job.

Hope some of that helps. IMHO, well worth the experiment to give it a shot and see if / how much it helps in your specific case.


Space Available
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours