Remove header while getting data through file stage

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
deesh
Participant
Posts: 193
Joined: Mon Oct 08, 2007 2:57 am

Remove header while getting data through file stage

Post by deesh »

Hi,

I want to be remove my file header(first 3 rows), please let me know how to remove header.
My file name contains like UKRIPR*.txt, if i use with file pattern option i can't utilize the ' tail -n +4' command.

above command works with only specific file option. let me know is there any other way.
rkashyap
Premium Member
Premium Member
Posts: 532
Joined: Fri Dec 02, 2011 12:02 pm
Location: Richmond VA

Post by rkashyap »

You can use External Source stage for solving this issue. Will you be removing header from a single file or from multiple files?

If from a single file, then set value of "Source Program" to
tail -n +4 <path>UKRIPR*.txt
and if from multiple files, then to
for i in <path>UKRIPR*.txt; do tail -n +4 $i; done
deesh
Participant
Posts: 193
Joined: Mon Oct 08, 2007 2:57 am

Post by deesh »

If i implemented that logic it's throwing the warnings like.

External_Source_14,0: Field "LAYER_LIM_AMT" has import error and no default value; data: {5 7 1 4 2 8 . 6 1 0d}, at offset: 209
External_Source_14,0: Import warning at record 0.
deesh
Participant
Posts: 193
Joined: Mon Oct 08, 2007 2:57 am

Post by deesh »

Yes it's working fine with the previous message logic mentioned logic


If from a single file, then set value of "Source Program" to
Quote:
tail -n +4 <path>UKRIPR*.txt


thanks
Post Reply