DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
olgc
Participant



Joined: 18 Nov 2003
Posts: 145

Points: 1684

Post Posted: Mon May 18, 2020 8:53 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Additional info: Ingestion Data datalake
Hi there,

How do you ingest your data into datalake: Datastage, Dataset, Informatica or sqoop, ...?

I were trying this for a while with Datastage, creating cases with both IBM and Cloudera. With JDBC driver, we can ingest small amount of records (fewer than 1000) successfully, but ran hours for just 10 thousand rows. A good news was it's very good of extracting from datalake, just like from any other data store.

Let us know how you do.

Thanks,
olgc
Participant



Joined: 18 Nov 2003
Posts: 145

Points: 1684

Post Posted: Mon May 18, 2020 11:49 am Reply with quote    Back to top    

To be clear, datalake is meant created with Hadoop technology: Hive, Impala, hBase, and/or Kudu.

Thanks,
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2890
Location: USA
Points: 21936

Post Posted: Thu May 21, 2020 11:24 am Reply with quote    Back to top    

Our place does not have any of those technologies, but have you tried the File connector stage?

https://www.ibm.com/support/knowledgecenter/SSZJPZ_11.7.0/com.ibm.swg.im.iis.conn.filecon.usage.doc/topics/filecon_parent.html

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
olgc
Participant



Joined: 18 Nov 2003
Posts: 145

Points: 1684

Post Posted: Mon May 25, 2020 1:19 pm Reply with quote    Back to top    

[quote="qt_ky"]Our place does not have any of those technologies, but have you tried the File connector stage?

Yes, we did, but unfortunately didn't get it worked due to permission issue (we set the highest security level possible for our Hadoop platform, so permission always a tough task).
We got it worked well outside of Datastage by sftp or Linux command scp. So one solution is creating the target as a file, and transfer the file to Hadoop platform, then use Impala / Hive load data statement load it into table.

Late we developed a better solution, get this easy and very productive. Please refer https://www.linkedin.com/pulse/datalake-ingestion-tool-small-simple-yet-powerful-scala-steven-huang/ for a glimpse of the solution.

Thanks,
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours