Issue while reading sequential file

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
rohit_mca2003
Participant
Posts: 41
Joined: Wed Oct 08, 2008 9:19 am

Issue while reading sequential file

Post by rohit_mca2003 »

Hi,

This is very common error but I did not get any suitable answer from other entries so posting this as new query.

I am trying to read csv file which has record delimiter as '\r\n'. It is windows file. I can see <CR><LF> in editors after each record.

Schema File has been defined as below:

record
{delim_string=',',charset='ASCII',final_delim=end,record_delim_string='\r\n',quote='"',null_field=''} (
Col1: string[max=20];
Col2:nullable string[max=1000];
Col3:nullable string[max=1000];
Col4:nullable string[max=1000];
)

DataStage job gives below warning and all the records in the file goes to reject.

Sequential_File_0,0: Input buffer overrun at field "Col3"
Sequential_File_0,0: Import unsuccessful at record 0.

But when I change the record_delim_string='\n' in schema file, then file processed successfully and I get all records in output.

Since this is windows file and have record delimiter as \r\n, so it should work when I define record_delim_string as '\r\n'.

Please help. Thanks.
Rohit
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Just out of curiosity, why are you trying to force the answer to be the one you want when you already have the answer that works? For the record, that newline character "\n" works for both UNIX and DOS, knowing which translates to one character and which translates to two. :wink:
-craig

"You can never have too many knives" -- Logan Nine Fingers
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Does your file transfer utility convert the DOS-style record delimiters to UNIX-style? Have you looked at the file on the UNIX machine?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
rohit_mca2003
Participant
Posts: 41
Joined: Wed Oct 08, 2008 9:19 am

Post by rohit_mca2003 »

To answer queries:

1. Actually we have mechanism to create schema file based on metadata of the file. Since metadata says this is DOS format file so schema file automatically takes '\r\n' as record delimiter string.
Other files from same source works fine.

2. I already checked file in UNIX and it has CRLF at the end of each record. As I mentioned that other files from same source does not have any problem. Connect Direct does not change the file format during transfer in this case.

Please suggest.

Thanks.
Rohit
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Re: Issue while reading sequential file

Post by ray.wurlod »

Why, then, is this true?
rohit_mca2003 wrote:when I change the record_delim_string='\n' in schema file, then file processed successfully and I get all records in output.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Do you have any kind of an update on this?
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply