[File Connector] Not delimited Files

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
joycerecacho
Participant
Posts: 298
Joined: Tue Aug 26, 2008 12:17 pm

[File Connector] Not delimited Files

Post by joycerecacho »

Hi everybody.

I am using 'File Connector' object to read txt files from HDFS. but some of them are not delimited, fields are positionals.

Example:
JoyceRecacho34 years oldBrasil

In other words, it would be like:

Field1: Row[1,12]
Field2: Row[13,12]
Field1: Row[25,6]

=======
I noticed that using the File Format like 'Implicit', some of the rows are split correctly however the majority of rows are disaligned, being droped.

Has anybody already passed through this issue?

As a workaround I read the file as it was a delimited one, and cut the fields like:
Field1: Row[1,12]
Field2: Row[13,12]
Field1: Row[25,6]

It worked fine, but the customer prefer the fields delimited conforming its definition at the sheet 'Column'.
Is there any way?
It reminds me the 'Sequential File' that we just inform "Record Length = Fixed" and "Delimiter = None" and it works.

Thanks in advance.

Best regards,
Joyce A. Recacho
São Paulo/SP
Brazil
UCDI
Premium Member
Premium Member
Posts: 383
Joined: Mon Mar 21, 2016 2:00 pm

Post by UCDI »

for fixed positions, left and right should do it. you can string these together eg
f1 = left(in, 12)
f2 = left(right(in,12),12)
etc
joycerecacho
Participant
Posts: 298
Joined: Tue Aug 26, 2008 12:17 pm

Post by joycerecacho »

Did you mean... in a transformer, as I did?
Cause I was thinking about solve it through File Connector only.
Joyce A. Recacho
São Paulo/SP
Brazil
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Try using a Sequential File stage, specify Fixed Width when importing the metadata, and specify the column widths during that process. Or edit the table definition details, and set the column Width property for each column.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

I know he'd do this since it's only three more words. Here's what is cut off:

"roperty for each column."
-craig

"You can never have too many knives" -- Logan Nine Fingers
joycerecacho
Participant
Posts: 298
Joined: Tue Aug 26, 2008 12:17 pm

Post by joycerecacho »

Oh nice, Ray.

I am still having problems.

Actually I read that you can configure the properties of the 'File Connector' the way it can understand the fixed width of the fields:

* File format: implicit
* Record definition: Delimited String
* Definition source: <name>:<definition>[length]
Between each field definition I tried ' ' or nothing ''
eg:
FieldName1:Char[1]FieldName2:Char[1]
FieldName1:Char[1] FieldName2:Char[1]

I might informed something wrong cause it didnt work.

Any suggestion?
Joyce A. Recacho
São Paulo/SP
Brazil
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Guessing IBM would know... <crosses fingers>
-craig

"You can never have too many knives" -- Logan Nine Fingers
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Try importing the Sequential File definition (not via the File Connector) then proceed as I earlier advised. You may need to share the result into the common metadata repository from DataStage Designer, but that's not difficult.
Last edited by ray.wurlod on Wed Feb 13, 2019 1:33 pm, edited 1 time in total.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
joycerecacho
Participant
Posts: 298
Joined: Tue Aug 26, 2008 12:17 pm

Post by joycerecacho »

Hi Ray,
Could you help me?

Tks in advance.
Joyce A. Recacho
São Paulo/SP
Brazil
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

You've got the important bits. However...

"r, but that's not difficult."
-craig

"You can never have too many knives" -- Logan Nine Fingers
joycerecacho
Participant
Posts: 298
Joined: Tue Aug 26, 2008 12:17 pm

Post by joycerecacho »

It didnt work here, so I let the solution like initially: I split the positional file cutting columns inside transformer.
Unfortunately, just a workaround for now.
Joyce A. Recacho
São Paulo/SP
Brazil
Post Reply