DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
sarathchandrakt
Participant



Joined: 29 Aug 2014
Posts: 45
Location: Mumbai
Points: 465

Post Posted: Wed May 08, 2019 10:46 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hi,

I am already using sed '1d;$d' to remove first and last line of the file. Now I have to remove carriage return from the file too. I have commands that can do both tasks separately. But I'm trying to find a command that can do all tasks in one filter.

Any help is appreciated.

Thanks.
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42964
Location: Denver, CO
Points: 221557

Post Posted: Thu May 09, 2019 6:26 pm Reply with quote    Back to top    

Can I ask the "why" of this? Is it because it's a Windows/DOS file? If so, there are other ways of handling it.

_________________
-craig

Now I've got that feeling once again, I can't explain; you would not understand
this is now who I am. I have become comfortably numb.
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54519
Location: Sydney, Australia
Points: 295643

Post Posted: Fri May 10, 2019 5:01 pm Reply with quote    Back to top    

You can probably adapt the sed script, but I'd use tr -d \n to remove newlines (or tr -d \r to remove carriage returns). Craig's question is relevant, though, because if it's only DOS-sty ...

_________________
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Rate this response:  
Not yet rated
mouthou
Participant



Joined: 04 Jul 2004
Posts: 208

Points: 1991

Post Posted: Sun May 12, 2019 4:15 pm Reply with quote    Back to top    

Based on Craig's and Ray's response focusing on carriage returns, I am slightly confused of your need. I think you are looking for a command pattern which does both header/trailer removal and CR removal in one shot. If so, this seems a unix command related question which is to be put in SeqFile stage option.

Did you try unix command piping something like " sed '1d;$d' <file> | tr -d ''. Another option of enclosing commands with && would also work like "(sed '1d;$d') && (tr -d )". Explore the exact syntaxt for the both the patterns, try running it from command prompt first and it should fix your need.
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42964
Location: Denver, CO
Points: 221557

Post Posted: Sun May 12, 2019 6:51 pm Reply with quote    Back to top    

It does seem that they are looking for such a "one shot" command. I was just trying to see if this CR was part of a DOS CR/LF record terminator string and if the command was (in essence) doing a DOS2UNIX terminator change to the UNIX LF. If that's the case, it's an unnecessary change as noted.

_________________
-craig

Now I've got that feeling once again, I can't explain; you would not understand
this is now who I am. I have become comfortably numb.
Rate this response:  
Not yet rated
sarathchandrakt
Participant



Joined: 29 Aug 2014
Posts: 45
Location: Mumbai
Points: 465

Post Posted: Mon May 13, 2019 1:05 am Reply with quote    Back to top    

Thank you everyone for the responses. I am getting extra CRs in file that we get from third party source. We figured it would be easy to fix it from our end.

I used TR in before job sub routineto remove CRs and then used sed in seq stage filter to remove header and footer. I was trying to accomplish both in one statement.
Rate this response:  
Not yet rated
mouthou
Participant



Joined: 04 Jul 2004
Posts: 208

Points: 1991

Post Posted: Wed May 15, 2019 12:12 pm Reply with quote    Back to top    

Any particular reason to go for a routine for CR removal? What is the issue in using unix in removing CRs and that too in the same place where header and trailer are handled.

Wondering what is made possible by a routine when unix can easily do the same.
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54519
Location: Sydney, Australia
Points: 295643

Post Posted: Wed May 15, 2019 5:19 pm Reply with quote    Back to top    

Probably using ExecSH as the before-job subroutine as a quick, easy interface to the UNIX command pipeline, thus avoiding the need for a sequence job.

_________________
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Rate this response:  
Not yet rated
UCDI



Group memberships:
Premium Members

Joined: 21 Mar 2016
Posts: 367

Points: 3806

Post Posted: Wed May 15, 2019 11:24 pm Reply with quote    Back to top    

perhaps less well know, you can call several unix commands back to back with a ; between them in the shell execute stage. here, piped stream is perhaps as good or better, but if you need it...

you can invoke the unix commands in a parallel job too, via a routine, if the before/after is insufficient.
Rate this response:  
Not yet rated
sarathchandrakt
Participant



Joined: 29 Aug 2014
Posts: 45
Location: Mumbai
Points: 465

Post Posted: Tue May 21, 2019 9:31 pm Reply with quote    Back to top    

The reason why we didn't run unix commands in sequence job is sometimes, we will be asked to process just a single file and running the sequence will trigger multiple jobs. So, we decided to keep the whole logic in the parallel job itself.

Honestly, I didn't think of using routines in parallel job. That is something I will defiantly consider in future. We were in a rush to do a quick fix to move to production.

Thanks again for all the input.
Rate this response:  
Not yet rated
mouthou
Participant



Joined: 04 Jul 2004
Posts: 208

Points: 1991

Post Posted: Wed May 22, 2019 1:18 pm Reply with quote    Back to top    

Just so you know. I don't think any of the response above gave a slight reference of using a Sequence as such. Those direct options were such that you could either put in ExecSH section as Ray mentioned or in Seq File stage directly.


Last edited by mouthou on Wed May 22, 2019 1:33 pm; edited 1 time in total
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours