DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
jneasy
Participant



Joined: 29 Jan 2012
Posts: 31
Location: Australia
Points: 313

Post Posted: Thu Feb 07, 2019 10:49 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Windows
Hi,

I was hoping that someone has had experience cataloging Hadoop assets in IGC and provide feedback on how easy/painful the process was.

Currently I am working at a site where they decided to install the minimum to use DataStage and QualityStage, even though the client is entitled to install the entire IIS suite (long story).

I am trying to put together some estimates on how long it would take to ingest some metadata and produce lineage reports. A core requirement is ingesting Hadoop assets such as Parquet files, Hive tables and views and Impala tables and views. Because the client has only installed DS and QS I cant even test if only importing the Parquet file, table definition and view definition is all IGC needs to then be able to generate a File --> Table --> View lineage report.

Has anyone had experience with ingestion and use of Hadoop assets in IGC and tell me how easy or difficult it is to implement?

Thanks,
Joe.
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3824

Points: 30832

Post Posted: Thu Feb 07, 2019 4:00 pm Reply with quote    Back to top    

There are a myriad of ways to accomplish this task, and probably many of them needed in combination. Some thoughts... Try to get them to implement IMAm. You canget to many of the Connectors via ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
jneasy
Participant



Joined: 29 Jan 2012
Posts: 31
Location: Australia
Points: 313

Post Posted: Sun Feb 10, 2019 10:40 am Reply with quote    Back to top    

I forgot to mention that we do have IMAM, and I have been able to import Hadoop tables and Files. However, without IGC i am unable to test if the lineage is automatically discovered between the Parquet files and the Table definition.
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54534
Location: Sydney, Australia
Points: 295710

Post Posted: Sun Feb 10, 2019 10:41 pm Reply with quote    Back to top    

Push harder for them to install IGC. Even if it remains unused as a tool, its REST API is singularly useful. And it's reasonably inexpensive, particularly compared to the current investment.

_________________
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours