DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
k.prashanthi
Participant



Joined: 08 Dec 2016
Posts: 16

Points: 155

Post Posted: Mon Jan 22, 2018 9:59 am Reply with quote    Back to top    

DataStage® Release: 9x
Job Type: Parallel
OS: Windows
Additional info: Sequential File
Hello,

I have a new requirement to work with comma delimited files with no quotes for blank values but I have never come across with this kind of files to work with. For example:

D,0,SB,,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
D,1,IN,,0,0,0,0,90000000,0,0,0,0,0,0,0,0,0,0
D,4,FEDOUT,,0,0,0,0,90000000,0,0,0,0,0,0,0
D,6,EB,,0,0,0,0,0,0
D,7,DEP,LOC,0,0,0,0,90000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,90000000

How to read this kind of files??

TIA.


Last edited by k.prashanthi on Wed Jan 24, 2018 2:10 pm; edited 2 times in total
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1237

Points: 8121

Post Posted: Mon Jan 22, 2018 10:07 am Reply with quote    Back to top    

Does the Sequential File stage with nullable columns not work for you?
Rate this response:  
Not yet rated
k.prashanthi
Participant



Joined: 08 Dec 2016
Posts: 16

Points: 155

Post Posted: Mon Jan 22, 2018 10:36 am Reply with quote    Back to top    

Hi Paul,

No. It did not work. Any other ideas?
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42574
Location: Denver, CO
Points: 219079

Post Posted: Mon Jan 22, 2018 12:46 pm Reply with quote    Back to top    

Specifics, please. What exactly "did not work" about it?

_________________
-craig

I know I don't say this enough, but I like when you talk to me. It's much better than when nobody talks to me. Or when people that I don't like will not stop talking to me.
Rate this response:  
Not yet rated
BillB



Group memberships:
Premium Members

Joined: 13 Nov 2007
Posts: 48

Points: 382

Post Posted: Mon Jan 22, 2018 4:32 pm Reply with quote    Back to top    

The server Sequential File stage will read a CSV file like your example. Set 'Outputs-Format-Missing columns action' to 'Map empty string', and 'Outputs-Columns-Incomplete Column' to 'Replace'.

If you need your CSV file to be an input to a parallel job, consider writing a pre-processor server job that reads your example CSV and writes a second CSV that becomes the parallel job input.
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42574
Location: Denver, CO
Points: 219079

Post Posted: Mon Jan 22, 2018 6:56 pm Reply with quote    Back to top    

Right, was going to get there. Wink

Another option, a Server Shared Container in your PX job that just handles the sequential file read.

_________________
-craig

I know I don't say this enough, but I like when you talk to me. It's much better than when nobody talks to me. Or when people that I don't like will not stop talking to me.
Rate this response:  
Not yet rated
k.prashanthi
Participant



Joined: 08 Dec 2016
Posts: 16

Points: 155

Post Posted: Wed Jan 24, 2018 12:54 pm Reply with quote    Back to top    

BillB,

That would work but we don't have server job as we have Data Quality license only. Any other ideas on this?
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1237

Points: 8121

Post Posted: Wed Jan 24, 2018 5:01 pm Reply with quote    Back to top    

if all else fails, you could modify the input file.

https://askubuntu.com/questions/20414/find-and-replace-text-within-a-file-using-commands

Command:
sed -i 's/original/new/g' file.txt


SO something like this:
sed -i 's/,,/,"",/g' file.txt
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54221
Location: Sydney, Australia
Points: 294071

Post Posted: Sun Jan 28, 2018 7:06 pm Reply with quote    Back to top    

And the sed command that Paul suggested could even be a Filter in the Sequential File stage.
Or executed via a before-job subroutine.

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours