Skip header record from complex flat file stage

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
harikhk
Participant
Posts: 64
Joined: Tue Jun 04, 2013 11:36 am

Skip header record from complex flat file stage

Post by harikhk »

Hi,

I am receiving zipped binary (EBCDIC))complex flat file
This is a fixed width file and contains a header record

Could you please suggest how an I filter the header record in the complex flat file stage

At the job level the file is being unzipped and on running the job, the job is failing with the below error


Short read encountered on import; this most likely indicates one of the following possibilities:
1) the import schema you specified is incorrect
2) invalid data (the schema is correct, but there is an error in the data).

I understand the issue is with file definition or with the data.

I was told to remove the header record and continue the loading.
Please suggest how can I get the header record filtered from complex flat file stage
Thanks,
HK
*Go GREEN..Save Earth*
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Presumably there is some kind of "record type" field. Use that to direct the header to a different output.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
harikhk
Participant
Posts: 64
Joined: Tue Jun 04, 2013 11:36 am

Post by harikhk »

Tried creating a new record defining a single column as character type equal to the length of the record but of no result
Thanks,
HK
*Go GREEN..Save Earth*
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

How exactly is the file being unzipped "at the job level"? If you are sending it to standard out then you should be able to pipe it to something like "sed" to remove the first record before the job consumes it.
-craig

"You can never have too many knives" -- Logan Nine Fingers
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

Not sure about the compression aspect. If the data is coming into CFF in the clear, it should not be an issue.

CFF requires you to define every record type on the input, but it does not require you to have an output link for every type. I have many files with header and trailer records, and I never need the trailer, so I don't code an output link for it.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

... or that. :wink:
-craig

"You can never have too many knives" -- Logan Nine Fingers
vmcburney
Participant
Posts: 3593
Joined: Thu Jan 23, 2003 5:25 pm
Location: Australia, Melbourne
Contact:

Post by vmcburney »

Is this a file generated by Cobol or from IMS? There should be a Cobol Definition File or Copy Book that DataStage can import to load up the required metadata to read the file. This would not just identify the record type, level and position of the header record but would also handle Cobol keywords like OCCURS and REDEFINES.
harikhk
Participant
Posts: 64
Joined: Tue Jun 04, 2013 11:36 am

Post by harikhk »

Thank you all for the replies. It has been identified that the copy book definition was outdated and I am waiting for the revised copy book
And hope it does not create any issues
Thanks,
HK
*Go GREEN..Save Earth*
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

That'll do it every time, and says a great deal about information governance in your organization. You should escalate the existence of this lack of communication through the stewardship community - as far as the CDO if necessary.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

It is often the case that there is a very different "method" of version control between platforms. Cobol systems generally have an embedded (and sometimes proprietary) method. We use Endevor, which both manages code between regions (dev, test, prod) and does automatic compiles. Interfacing that with a DataStage environment, at least here, just doesn't work.

We download copybooks during development, then rely on that same communication that didn't work for HK. We also enjoy strong discipline in our host development, and changes to copybooks are almost always made in the end "filler" of a copybook layout. This means that our code only needs to change if we have data being added in that filler area. If we don't need it, it doesn't actually change for us.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
Post Reply