Multiple Cobol Copy Books

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
neeraj
Participant
Posts: 107
Joined: Tue May 24, 2005 4:09 am

Multiple Cobol Copy Books

Post by neeraj »

Hello,

I have 50 Cobol copy books(EBCDIC Format) that need to be populated into Teradata with minimal Transformation. The Job design would be

CFF ----> Modify/Transformer--->TD

One option is to create the 50 Datastage Jobs(read and load).

But I am looking for some generic option Like create Multiinstance jobs. But I believe CFF does't support schema files at run time.

Incase anyone is aware about any generic solution, please let me know.

Regards
neeraj
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Define "minimal transformation" for us, please.
-craig

"You can never have too many knives" -- Logan Nine Fingers
nikhil_bhasin
Participant
Posts: 50
Joined: Tue Jan 19, 2010 4:14 am

Post by nikhil_bhasin »

Maybe you can write a script to convert ebcdic files into ascii and then use the files in your job
vmcburney
Participant
Posts: 3593
Joined: Thu Jan 23, 2003 5:25 pm
Location: Australia, Melbourne
Contact:

Post by vmcburney »

This is very hard to do as the Complex Flat File stage does not support column propagation and dynamic ETL. Cobol mainframe files each have a copybook that describes them and this tends to be unique for each file. You will need at least 50 DataStage jobs - one per unique copybook. You may be populating more than 50 target tables - those copybooks that have multiple arrays and different record types can have many types of outputs.

You are going from complex data to flattened data - the flattening process may mean you have many more than 50 target tables.

Minimum transformation can also be tricky when it comes to handling blanks or nulls or empty fields, sometimes the CFF stage needs to convert these to a default value which is a type of transformation.

I have found that if the copybooks are accurate you can load a copybook to Teradata in under an hour but if there are formatting and consistency and data type problems in those copybooks it could take several days to debug a single file.
Post Reply