DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
olgc
Participant



Joined: 18 Nov 2003
Posts: 142

Points: 1656

Post Posted: Sun Jun 16, 2019 10:23 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hello every one, does any one ingest data to Hadoop platform (Hive, Impala) or Apache Kudu? How do you resolve the performance issue? We use Datastage JDBC connector with Cloudera Impala JDBC driver, it works well for extraction, but not loading - the insert performance is terrible: 20,000 records takes almost 8 minutes. Has any one there trying ingesting data to Hadoop platform/Apache Kudu? What's your experience?

Thanks,
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3821

Points: 30806

Post Posted: Mon Jun 17, 2019 9:13 am Reply with quote    Back to top    

Hi. I don't have any experience with Kudo, but in past threads here and elsewhere over the years, I've heard many people talk about loading to their hive and hive related tables using direct writes ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
olgc
Participant



Joined: 18 Nov 2003
Posts: 142

Points: 1656

Post Posted: Thu Jun 20, 2019 2:20 pm Reply with quote    Back to top    

Thanks, eostic. Yes, we try use file connector uploading result file to hdfs file system, then through there to Kudu. It'w a work around, not look that good.
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours