DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
TNZL_BI



Group memberships:
Premium Members

Joined: 20 Aug 2012
Posts: 24
Location: NZ
Points: 248

Post Posted: Mon Apr 10, 2017 10:43 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hi All ,

I have recently developed a job to connect to the Hive database in the Hadoop Ecosystem. Now I have used two methods to connect to the Hive database which are :-

1. ODBC Connector
2. Hive Connector

However , I am facing massive performance issues with the hive connector stage . Its taking hours to simply load some 80k rows where as when I use the ODBC connector stage , the performance is very good. We see this getting loaded in around 5 minutes time.

Does any one have an idea on this. Ideally the native connector stage should be faster and should have more options but in my case , the performance is really bad ...

Any inputs here will be very helpful.
TNZL_BI



Group memberships:
Premium Members

Joined: 20 Aug 2012
Posts: 24
Location: NZ
Points: 248

Post Posted: Sun Apr 30, 2017 6:00 pm Reply with quote    Back to top    

I have just got some patches to be installed on my services / engine tier as suggested by IBM . This may improve the speed. Will do that and then revert back with my findings
Rate this response:  
Not yet rated
AnnDSX
Participant



Joined: 04 Dec 2017
Posts: 4

Points: 41

Post Posted: Mon Mar 05, 2018 5:45 am Reply with quote    Back to top    

Hello,

Did you install the patches and see performance enhancement

Thanks
Rate this response:  
Not yet rated
rkashyap



Group memberships:
Premium Members

Joined: 02 Dec 2011
Posts: 518
Location: Richmond VA
Points: 4665

Post Posted: Mon Mar 05, 2018 4:16 pm Reply with quote    Back to top    

Hive connector leverages JDBC connectivity.

We are using both ODBC Connector and Hive Connector for connect with Hive and have not seen much difference between the performance of the two.
Rate this response:  
Not yet rated
AnnDSX
Participant



Joined: 04 Dec 2017
Posts: 4

Points: 41

Post Posted: Mon Mar 05, 2018 10:58 pm Reply with quote    Back to top    

We are using the FileConnector for moving the files to HDFS and the performance is fair. However the performance of Hive connector is dismal.

The best that we could achieve was writing 1000 records in 20 minutes.
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours