Sequential file CR/LF working differently in AIX vs LINUX

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
hsahay
Premium Member
Premium Member
Posts: 175
Joined: Wed Mar 21, 2007 9:35 am

Sequential file CR/LF working differently in AIX vs LINUX

Post by hsahay »

Hi
We just migrated our datastage installation from AIX to LINUX and we have run into an issue with the sequential file stage. The sequential file stage is used to read a file and store the fields in an oracle table.

The sequential file is reading files that may have CRLF or just LF as record delimited.

In AIX, the sequential file stage, with the FINAL_DELIMITER set to END, is able to strip both CRLF and LF from the last field before storing it into a DB.

But under LINUX, it is only stripping the LF from CRLF leaving the CR as part of the last field which then gets stored in the DB along with the field value as a non printable character.

Short of changing the jobs and the scripts, what else can we do to make LINUX behave more like AIX where without any prompting it is able to understand that CRLF or LF are both record terminators and need to be stripped.
vishal
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

You can get it right on a stage-by-stage basis (setting the Record Delimiter or Record Delimiter String property as appropriate).
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

... unless you are saying they can be different inside the same file? Or do you just mean it can change per file but all will be consistent within any given file? :?

Just trying to understand the scope of the issue.
-craig

"You can never have too many knives" -- Logan Nine Fingers
hsahay
Premium Member
Premium Member
Posts: 175
Joined: Wed Mar 21, 2007 9:35 am

Post by hsahay »

Thanks for your responses guys but this is a strange problem

To answer your question Chulett, Every file will either have a CRLF or just LF. But we can't predict which file will have which line delimiter.

The issue is that when we had these jobs running in AIX environment, the sequential file stage, with its FINAL_DELIMITER=END, was somehow able to correctly process both kinds of files.

When a file had CRLF, it removed the CRLF from the last field and when it had LF, it removed the LF.

But now since we moved everything to LINUX, it is only removing LF from both kinds of files leaving CR untouched which is now ending up in the database as part of the last field.

Quesiton - What was making AIX handle both CRLF and LF correctly.
And how to we make LINUX behave in a similar manner.
vishal
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

AIX is... special. And I sincerely doubt you'll be able to make anything else behave like that. Perhaps some sort of "pre-processing" is in order?
-craig

"You can never have too many knives" -- Logan Nine Fingers
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

Just a thought...

When you migrated from AIX to Linux did you also upgrade your DataStage version?

Newer versions of DataStage tend to get more strict, so it could be due to that rather than the OS.
Choose a job you love, and you will never have to work a day in your life. - Confucius
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Good point. I shiver in antici .... .... ... pation!
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply