DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
AGStafford



Group memberships:
Premium Members

Joined: 16 Jan 2003
Posts: 30

Points: 306

Post Posted: Thu May 10, 2018 7:41 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Additional info: OpenIGC
Has anyone added Mainframe Jobs, Mainframe files, Shell Scripts and Scheduler objects into the IGC for lineage?

We have mainframe jobs that create files on the Mainframe that are then Ftp'd down to unix, where shell scripts move/process them before calling DataStage jobs to load them into tables.

Our business is really interested in getting the lineage all the way back to the mainframe.
We are probably able to generate information that could be loaded into the IGC for all those objects, but I just don't quite get how to build the objects.

Using information from https://dsrealtime.wordpress.com/2015/07/29/open-igc-is-here/ and IBM I could probably build the objects, but linking them together seems to be outside my understanding.
Especially being able to link to the actual operational Metadata values from the job runs.
metamata



Group memberships:
Premium Members

Joined: 02 Nov 2017
Posts: 2

Points: 19

Post Posted: Mon Jul 09, 2018 11:57 am Reply with quote    Back to top    

We have similar requirement as well. Mainframe system sends File1 in EBCDIC format to DS Engine Server and is read by DS jobs using CFF stage. We created an Extension Mapping Document for the mainframe side logic that creates File1. DS jobs use Parameterized File names to read the file from mainframe. We enabled Operational Metadata while running the DS jobs and the OMD is getting copied over to the Repository. The question is how to link the output of Extension Mapping Document to the input of the DS job?
Rate this response:  
Not yet rated
asorrell
Site Admin

Group memberships:
Premium Members, DSXchange Team, Inner Circle, Server to Parallel Transition Group

Joined: 04 Apr 2003
Posts: 1681
Location: Colleyville, Texas
Points: 22831

Post Posted: Mon Jul 09, 2018 2:19 pm Reply with quote    Back to top    

The target of the extension mapping (The mainframe file) should be imported into the repository using IMAM, as an Implemented data resource. Then it should show up in the lineage when DataStage processes that data resource.

_________________
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2017
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3781

Points: 30365

Post Posted: Wed Jul 11, 2018 1:23 am Reply with quote    Back to top    

Please provide more details on the questions you have or the problems you are running into.... OpenIGC provides a good framework for defining your processes (COBOL programs, batch JCL, etc.) and you ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
metamata



Group memberships:
Premium Members

Joined: 02 Nov 2017
Posts: 2

Points: 19

Post Posted: Mon Jul 16, 2018 12:13 pm Reply with quote    Back to top    

asorrell,
The issue seems to be the context of the EMD output (folder) is not matching the context of the imported the mainframe file 1 using IMAM (Data File on engine server host). The mainframe file is read by our Datastage job using CFF stage and parameterized file name. We did run the DS job with OMD turned on and the OMD is getting copied over to the repository. Per IBM the stitching of EMD output to input file of CFF stage is not available in 11.5 and works from 11.7. For now the solution provided by IBM for 11.5 FP1, is to create an "Application" for the EMD output file, then map the output values of the application to the input fields for the file read by CFF stage. With this we are able to see the data lineage that is stitched together.

Ernie,
Any thoughts? Is there is a better way using OpenIGC?

Thanks!
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3781

Points: 30365

Post Posted: Mon Jul 16, 2018 6:19 pm Reply with quote    Back to top    

Hi. It can be done without an application data source. First, if you are using EMDs, create one manually in the GUI and then export it. Use the gui for selecting source or target and p ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours