Recover first line for each file treated by the same job

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
auvray.muriel
Participant
Posts: 43
Joined: Wed Feb 19, 2003 7:17 am
Location: France - Paris
Contact:

Recover first line for each file treated by the same job

Post by auvray.muriel »

Hello,

I have several files sequentiels in a repertory, and a job which treats the whole of the files of the repertory.

Each one of these files contain a first recording different from the others (cutting different).

Code: Select all

File_1 INPUT
Line 1 : AAA12345ZZZZZZZZZZ
Line2 : 38890BBBBBCCCDDDDD
Line2 : 99990EEEEEESSSVVVVV

File_2 INPUT
Line 1 : XXX88888ZZZZZZZZZZ
Line2 : 38890TTTTTUUUDDDDD
Line2 : 99990AAAARRRGGGGGG

OUTPUT
1234538890BBBBBCCCDDDDD
1234599990EEEEEESSSVVVVV
8888838890TTTTTUUUDDDDD
8888899990AAAARRRGGGGGG


I must recover information in the first line of each file. I make a test:

If @INROWNUM = 1 then I recover the value in the first line, and I store it in a stage variable.

This test functions for the first file, but as soon as the job treats the following files of the repertory, variable @INROWNUM is not re-initialized, and my test does not go any more.

Somebody have a solution?

One spoke to me about the columns generator, but I do not know how to count the lines with this stage.




Thank you very much
kduke
Charter Member
Charter Member
Posts: 5227
Joined: Thu May 29, 2003 9:47 am
Location: Dallas, TX
Contact:

Post by kduke »

Why not use head and tail to cut the first line off. If you do a search this has been covered before. If you cannot find it then let us know.
Mamu Kim
vmcburney
Participant
Posts: 3593
Joined: Thu Jan 23, 2003 5:25 pm
Location: Australia, Melbourne
Contact:

Post by vmcburney »

I noticed your header row starts with a character while your data rows start with a number. You could check the first character of the input line and if it is not between 0 and 9 you can assume it is a heading row.
auvray.muriel
Participant
Posts: 43
Joined: Wed Feb 19, 2003 7:17 am
Location: France - Paris
Contact:

Post by auvray.muriel »

Thank kduke,

But how to make a head on each file, since when the job traverses the repertory, it makes the equivalent of a concatenation of file, therefore I do not find any more the first line starting from the 2nd file ?

vmcburney, it is a coincidence the fact that the first character is numerical. that will not be true forcing.

Muriel
kduke
Charter Member
Charter Member
Posts: 5227
Joined: Thu May 29, 2003 9:47 am
Location: Dallas, TX
Contact:

Post by kduke »

Put them in a different directory. Say your files all end in txt. Here is a shell script. You need to setup the directory shell variables and also trap whether the second directory exists.

Code: Select all

if [ -f $TMPDIR ]
then
  cd $TMPDIR
  rm *
else
  echo "Error: $TMPDIR does not exist"
  exit 1
fi
cd $FILEDIR
for FILENAME in *txt
do
  head -1 $FILENAME > $TMPDIR/$FILENAME
done
Mamu Kim
Post Reply