DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
dsuser_cai



Group memberships:
Premium Members, Heartland Usergroup

Joined: 13 Feb 2009
Posts: 151

Points: 1200

Post Posted: Tue Sep 12, 2017 10:47 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Additional info: I have few questions about how BI 11.5 works on hadoop
Hi

We have BI 11.5 on Linux, Hortonworks Data platform. We are building a huge data lake in hadoop. I have few questions about how BI 11.5 works. I couldnt find answers to these in google. So could you please share some light.

1. Hive Queries - If i want to run any HIVE query on interactive mode using PUTTY, by default the execution engine is on Tez, but we can change this to use map reduce engine. But how does BigIntegrate works? For Example when i use a HIVE stage does it runs on Tez or MR engine? What about if i use File Connector stage to build a HIVE table?
2. Dynamic configuration File - Can someone provide the IBM link for some details on what is Dynamic config file and how it differs from traditional DS config file?
3. is there a way to start a tez session and keep it open, then run all my HIVE queries and then once i complete all HIVE queries, i would like to close the tez session. I would like to do this in a BI JOB. for example... I want to open the tez session using an execute command stage (shell script), then run all my jobs that uses HIVE load and extractions, once complete then finally close tez session. I dont know if BI uses tez in the background, but when i run H-SQL's it runs on tez... so i'm assuming that BI would also use tez in the background. but i may be completely wrong..

So any advice on these would be very helpful.

_________________
Thanks
Karthick
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours