DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
neeraj
Participant



Joined: 24 May 2005
Posts: 107

Points: 1462

Post Posted: Mon Apr 29, 2013 1:16 am Reply with quote    Back to top    

DataStage® Release: 8x
Job Type: Parallel
OS: Unix
Additional info: Reading multiple mainframe files
Hello,

I have 50 Cobol copy books(EBCDIC Format) that need to be populated into Teradata with minimal Transformation. The Job design would be

CFF ----> Modify/Transformer--->TD

One option is to create the 50 Datastage Jobs(read and load).

But I am looking for some generic option Like create Multiinstance jobs. But I believe CFF does't support schema files at run time.

Incase anyone is aware about any generic solution, please let me know.

Regards
neeraj
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43085
Location: Denver, CO
Points: 222463

Post Posted: Mon Apr 29, 2013 7:24 am Reply with quote    Back to top    

Define "minimal transformation" for us, please.

_________________
-craig

"You can never have too many knives" -- Logan Nine Fingers
Rate this response:  
Not yet rated
nikhil_bhasin
Participant



Joined: 19 Jan 2010
Posts: 50

Points: 393

Post Posted: Tue Apr 30, 2013 3:55 am Reply with quote    Back to top    

Maybe you can write a script to convert ebcdic files into ascii and then use the files in your job
Rate this response:  
Not yet rated
vmcburney

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup

Joined: 23 Jan 2003
Posts: 3593
Location: Australia, Melbourne
Points: 28136

Post Posted: Tue Apr 30, 2013 5:33 pm Reply with quote    Back to top    

This is very hard to do as the Complex Flat File stage does not support column propagation and dynamic ETL. Cobol mainframe files each have a copybook that describes them and this tends to be unique for each file. You will need at least 50 DataStage jobs - one per unique copybook. You may be populating more than 50 target tables - those copybooks that have multiple arrays and different record types can have many types of outputs.

You are going from complex data to flattened data - the flattening process may mean you have many more than 50 target tables.

Minimum transformation can also be tricky when it comes to handling blanks or nulls or empty fields, sometimes the CFF stage needs to convert these to a default value which is a type of transformation.

I have found that if the copybooks are accurate you can load a copybook to Teradata in under an hour but if there are formatting and consistency and data type problems in those copybooks it could take several days to debug a single file.

_________________
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn: Vincent McBurney LinkedIn
Rate this response:  
Not yet rated
Lucas97
Participant



Joined: 24 Oct 2020
Posts: 1

Points: 6

Post Posted: Sat Oct 24, 2020 9:03 pm Reply with quote    Back to top    

This topic has some technical stuff being shared here with those who are up for this and want to know what it was all about. I cannot say much becuase maybe you will find a bit more about papers writers so you need to do some research in this manner sooner.
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54621
Location: Sydney, Australia
Points: 296179

Post Posted: Tue Oct 27, 2020 5:43 am Reply with quote    Back to top    

[Moderator: please remove spam message]

_________________
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Rate this response:  
Not yet rated
bruh
Participant



Joined: 09 Jan 2021
Posts: 3

Points: 17

Post Posted: Sat Jan 09, 2021 6:13 pm Reply with quote    Back to top    

Data storaging is a good way to keep in mind everything that was made. I know that my mates working at the top resume writing services also using this storage method. I think it is quite suitable for such organizations. So, also thank you for your post pal.
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54621
Location: Sydney, Australia
Points: 296179

Post Posted: Fri Feb 12, 2021 3:15 am Reply with quote    Back to top    

[Moderator: please remove spam message]

_________________
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours