Missing record delimiter "\n", saw EOF instead

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
faujong
Participant
Posts: 11
Joined: Fri Aug 17, 2018 9:03 am

Missing record delimiter "\n", saw EOF instead

Post by faujong »

Our DataStage parallel job has a Sequential File stage that reads a CSV file.
This is a sample of the file content. Each row ends with LF (Line Feed - hex = 0A), except the last row has no LF.

"Name","Currency","Description"LF
"ABC","USD","ABC Description"LF
"DEF","USD","DEF Description"

This is in Output > Properties:
Options:
First line is column names = True

This is in Output > Format:
Record level:
Final delimiter = end

Field defaults:
Delimiter = comma
Quote = double

When I run the DataStage, it imported all records successfully, but it gives me 2 warnings:
Warning: Missing record delimiter "\n", saw EOF instead
Warning: Import warning at record 81.
Import complete; 82 records imported successfully, 0 rejected.

There are 83 rows in the CSV file, with the 1st row as the column names.
The "Import warning at record ..." is always on the number of rows - 2, for example if the number of rows = 8, the Import warning will be at record 6.

It looks like the problem is because all rows ends with Line Feed, except for the last row (it doesn't end with Line Feed).
I have tried all the other Final delimiter (Record Level), but doesn't work.

Can I fix the warning ?
How can I read a CSV file who has all rows end with Line Feed, except for the last row ?

Thank you.
faujong
Participant
Posts: 11
Joined: Fri Aug 17, 2018 9:03 am

Re: Missing record delimiter "\n", saw EOF instead

Post by faujong »

I fixed the warning by adding a Line Feed at the end of the file using Before Job subroutine EXECSH sed -i -e '$a\' /dev/inbound/MyFile_*.csv
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Hopefully, someone will come along and let us know if there is or is not an actual solution with DataStage rather than having to employ the work around you posted. For completeness. :wink:
-craig

"You can never have too many knives" -- Logan Nine Fingers
faujong
Participant
Posts: 11
Joined: Fri Aug 17, 2018 9:03 am

Post by faujong »

Agree
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

A bit complex a solution, but it is a solution.

Perhaps easier would have been

Code: Select all

echo \n >> filename
via the ExecSH before-job subroutine.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

So... there's no option or property within the stage itself to handle this situation?
-craig

"You can never have too many knives" -- Logan Nine Fingers
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Not in the parallel Sequential File stage.

The server Sequential File stage has it handled and, as noted elsewhere here, could be encapsulated in a server Shared Container to be included in a parallel job.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
faujong
Participant
Posts: 11
Joined: Fri Aug 17, 2018 9:03 am

Post by faujong »

Thank you Ray
Sumith
Participant
Posts: 5
Joined: Tue Sep 04, 2018 8:37 am

Re: Missing record delimiter "\n", saw EOF instead

Post by Sumith »

I had similar kind of issue and one of the below functions. Please refer
ereplace(ereplace(echoProcDate.$CommandOutput,@FM,"")
CONVERT(Char(13),"",lnk.field)
Thanks
Sumith
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Such an approach is only appropriate if one can read the data in the first place. In the current case, the lack of a linefeed character prevented the last line of the file from being read. :cry:
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply