Hi all,
I have a sequential file with Pipe delimited format, and i have layout for that. But the some of the input records are not following the Metadata layout, i would like to write those records to another file and good records to good file.
Good Input record with correct lay out:-
AAA|BBB|CCC|123456| | |XXXXX|YYYYY|ZZZZZ|
bad records are
AAA|BBB|CCC|123456| |XXXX
bad records are ending or missing layout....
need to write those in different file.
I tried using 'Discard and Warn' - Incomplete column in Input layout
Any help is appreciated!!
Good and Bad records
Moderators: chulett, rschirm, roy
Titto,
I would recommend reading the sequential file as a 1 column file with no delimiters and in the first transform stage use the result of DCOUNT(In.BigColumn,'|') to see if the number of columns is correct and use the constraint to filter out badly formed records. You can then write the good records to a sequential file. This is then read (you should use the same sequential file stage in your job to ensure sequential processesing) with the same format as your current job works. If you need performance increases, then make this interim stage into a pipe type file for input and output and the data will not be written to or read from the disk.
I would recommend reading the sequential file as a 1 column file with no delimiters and in the first transform stage use the result of DCOUNT(In.BigColumn,'|') to see if the number of columns is correct and use the constraint to filter out badly formed records. You can then write the good records to a sequential file. This is then read (you should use the same sequential file stage in your job to ensure sequential processesing) with the same format as your current job works. If you need performance increases, then make this interim stage into a pipe type file for input and output and the data will not be written to or read from the disk.
</a>Hi ArndW
If i understood correctly,
first i need to read the source file with out any delimeters... and in the transformer use Dcount to get the number of delimeters (for example if i get the count of delemeters 50 then in the constraint Dcount(Link.rec,'|') = 50 then good records) and write to output file and use this output file as source to next transformer to write into a correct layout format.
I did the first part to get the Dcount from the main source file and got the 50 count.... then i used that number in constraint to write the records only with matches that value but it is writing all the records... Sure i am doing something wrong.
am i missing something!!!
Thanks!!!
If i understood correctly,
first i need to read the source file with out any delimeters... and in the transformer use Dcount to get the number of delimeters (for example if i get the count of delemeters 50 then in the constraint Dcount(Link.rec,'|') = 50 then good records) and write to output file and use this output file as source to next transformer to write into a correct layout format.
I did the first part to get the Dcount from the main source file and got the 50 count.... then i used that number in constraint to write the records only with matches that value but it is writing all the records... Sure i am doing something wrong.
am i missing something!!!
Thanks!!!
-
ameyvaidya
- Charter Member
- Posts: 166
- Joined: Wed Mar 16, 2005 6:52 am
- Location: Mumbai, India
Hi Titto,
Please go through the following topic. U could adapt the info present there for your design:
viewtopic.php?t=95873&highlight=
IHTH
Please go through the following topic. U could adapt the info present there for your design:
viewtopic.php?t=95873&highlight=
IHTH
Amey Vaidya<i>
I am rarely happier than when spending an entire day programming my computer to perform automatically a task that it would otherwise take me a good ten seconds to do by hand.</i>
<i>- Douglas Adams</i>
I am rarely happier than when spending an entire day programming my computer to perform automatically a task that it would otherwise take me a good ten seconds to do by hand.</i>
<i>- Douglas Adams</i>