DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
hsahay



Group memberships:
Premium Members

Joined: 21 Mar 2007
Posts: 174

Points: 2345

Post Posted: Mon Dec 18, 2017 11:43 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hi
We just migrated our datastage installation from AIX to LINUX and we have run into an issue with the sequential file stage. The sequential file stage is used to read a file and store the fields in an oracle table.

The sequential file is reading files that may have CRLF or just LF as record delimited.

In AIX, the sequential file stage, with the FINAL_DELIMITER set to END, is able to strip both CRLF and LF from the last field before storing it into a DB.

But under LINUX, it is only stripping the LF from CRLF leaving the CR as part of the last field which then gets stored in the DB along with the field value as a non printable character.

Short of changing the jobs and the scripts, what else can we do to make LINUX behave more like AIX where without any prompting it is able to understand that CRLF or LF are both record terminators and need to be stripped.

_________________
vishal
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54221
Location: Sydney, Australia
Points: 294071

Post Posted: Mon Dec 18, 2017 4:28 pm Reply with quote    Back to top    

You can get it right on a stage-by-stage basis (setting the Record Delimiter or Record Delimiter String property as appropriate).

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne
Rate this response:  
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42576
Location: Denver, CO
Points: 219102

Post Posted: Mon Dec 18, 2017 5:38 pm Reply with quote    Back to top    

... unless you are saying they can be different inside the same file? Or do you just mean it can change per file but all will be consistent within any given file? Confused

Just trying to understand the scope of the issue.

_________________
-craig

I know I don't say this enough, but I like when you talk to me. It's much better than when nobody talks to me. Or when people that I don't like will not stop talking to me.
Rate this response:  
hsahay



Group memberships:
Premium Members

Joined: 21 Mar 2007
Posts: 174

Points: 2345

Post Posted: Tue Dec 19, 2017 1:42 pm Reply with quote    Back to top    

Thanks for your responses guys but this is a strange problem

To answer your question Chulett, Every file will either have a CRLF or just LF. But we can't predict which file will have which line delimiter.

The issue is that when we had these jobs running in AIX environment, the sequential file stage, with its FINAL_DELIMITER=END, was somehow able to correctly process both kinds of files.

When a file had CRLF, it removed the CRLF from the last field and when it had LF, it removed the LF.

But now since we moved everything to LINUX, it is only removing LF from both kinds of files leaving CR untouched which is now ending up in the database as part of the last field.

Quesiton - What was making AIX handle both CRLF and LF correctly.
And how to we make LINUX behave in a similar manner.

_________________
vishal
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42576
Location: Denver, CO
Points: 219102

Post Posted: Tue Dec 19, 2017 2:13 pm Reply with quote    Back to top    

AIX is... special. And I sincerely doubt you'll be able to make anything else behave like that. Perhaps some sort of "pre-processing" is in order?

_________________
-craig

I know I don't say this enough, but I like when you talk to me. It's much better than when nobody talks to me. Or when people that I don't like will not stop talking to me.
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2768
Location: USA
Points: 20827

Post Posted: Tue Dec 19, 2017 2:51 pm Reply with quote    Back to top    

Just a thought...

When you migrated from AIX to Linux did you also upgrade your DataStage version?

Newer versions of DataStage tend to get more strict, so it could be due to that rather than the OS.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42576
Location: Denver, CO
Points: 219102

Post Posted: Tue Dec 19, 2017 6:30 pm Reply with quote    Back to top    

Good point. I shiver in antici .... .... ... pation!

_________________
-craig

I know I don't say this enough, but I like when you talk to me. It's much better than when nobody talks to me. Or when people that I don't like will not stop talking to me.
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours