DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic is not resolved, but there is a WORKAROUND.
Author Message
faujong
Participant



Joined: 17 Aug 2018
Posts: 11

Points: 158

Post Posted: Thu Aug 30, 2018 12:29 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Our DataStage parallel job has a Sequential File stage that reads a CSV file.
This is a sample of the file content. Each row ends with LF (Line Feed - hex = 0A), except the last row has no LF.

"Name","Currency","Description"LF
"ABC","USD","ABC Description"LF
"DEF","USD","DEF Description"

This is in Output > Properties:
Options:
First line is column names = True

This is in Output > Format:
Record level:
Final delimiter = end

Field defaults:
Delimiter = comma
Quote = double

When I run the DataStage, it imported all records successfully, but it gives me 2 warnings:
Warning: Missing record delimiter "\n", saw EOF instead
Warning: Import warning at record 81.
Import complete; 82 records imported successfully, 0 rejected.

There are 83 rows in the CSV file, with the 1st row as the column names.
The "Import warning at record ..." is always on the number of rows - 2, for example if the number of rows = 8, the Import warning will be at record 6.

It looks like the problem is because all rows ends with Line Feed, except for the last row (it doesn't end with Line Feed).
I have tried all the other Final delimiter (Record Level), but doesn't work.

Can I fix the warning ?
How can I read a CSV file who has all rows end with Line Feed, except for the last row ?

Thank you.
faujong
Participant



Joined: 17 Aug 2018
Posts: 11

Points: 158

Post Posted: Thu Aug 30, 2018 2:24 pm Reply with quote    Back to top    

I fixed the warning by adding a Line Feed at the end of the file using Before Job subroutine EXECSH sed -i -e '$a\' /dev/inbound/MyFile_*.csv
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42790
Location: Denver, CO
Points: 220550

Post Posted: Fri Aug 31, 2018 10:25 am Reply with quote    Back to top    

Hopefully, someone will come along and let us know if there is or is not an actual solution with DataStage rather than having to employ the work around you posted. For completeness. Wink

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
faujong
Participant



Joined: 17 Aug 2018
Posts: 11

Points: 158

Post Posted: Fri Aug 31, 2018 11:31 am Reply with quote    Back to top    

Agree
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54407
Location: Sydney, Australia
Points: 295092

Post Posted: Mon Sep 03, 2018 9:31 am Reply with quote    Back to top    

A bit complex a solution, but it is a solution.

Perhaps easier would have been
Code:
echo \n >> filename
via the ExecSH before-job subroutine.

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42790
Location: Denver, CO
Points: 220550

Post Posted: Mon Sep 03, 2018 1:04 pm Reply with quote    Back to top    

So... there's no option or property within the stage itself to handle this situation?

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54407
Location: Sydney, Australia
Points: 295092

Post Posted: Mon Sep 03, 2018 8:58 pm Reply with quote    Back to top    

Not in the parallel Sequential File stage. The server Sequential File stage has it handled and, as noted elsewhere here, could be encapsulated in a server Shared Container to be included in a par ...

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
faujong
Participant



Joined: 17 Aug 2018
Posts: 11

Points: 158

Post Posted: Tue Sep 04, 2018 8:24 am Reply with quote    Back to top    

Thank you Ray
Rate this response:  
Not yet rated
Sumith
Participant



Joined: 04 Sep 2018
Posts: 5

Points: 31

Post Posted: Thu Sep 06, 2018 2:59 pm Reply with quote    Back to top    

I had similar kind of issue and one of the below functions. Please refer
ereplace(ereplace(echoProcDate.$CommandOutput,@FM,"")
CONVERT(Char(13),"",lnk.field)

_________________
Thanks
Sumith
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54407
Location: Sydney, Australia
Points: 295092

Post Posted: Thu Sep 06, 2018 5:43 pm Reply with quote    Back to top    

Such an approach is only appropriate if one can read the data in the first place. In the current case, the lack of a linefeed character prevented the last line of the file from being read. Crying or Very sad

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours