CFF Stage and special or non-printable characters (again)

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
ffsw13
Premium Member
Premium Member
Posts: 31
Joined: Mon Feb 01, 2016 10:29 am
Location: USA
Contact:

CFF Stage and special or non-printable characters (again)

Post by ffsw13 »

Greetings,

I've been saddled with a group of DataStage EE and Server jobs that are suffering of the same aliment; seeing special or non-printable characters in various fields in the source records, and can't find a way to eradicate them once in DataStage. I've read the various posts in this forum on the topic but can't say I came away with a clear answer, and not being a recent Mainframe guy (programmed in COBOL a few decades ago on non-MF systems), not sure what path to go down.

Anyway, let me get on with it. to illustrate an example of the problem I'm having, I have the following sample copybook layout:

7 columns - level 2 - DISPLAY_NUMERIC
1 column - level 2 - GROUP
3 columns - level 3 - DISPLAY_NUMERIC
1 column - level 2 - GROUP
2 columns - level 3 - DISPLAY_NUMERIC
1 column - level 2 - DECIMAL - length 5
5 columns - level 2 - CHARACTER
1 column - level 2 - DECIMAL - length 7 Scale 4
4 columns - level 2 - CHARACTER
problem col - level 2 - CHARACTER - length 12 - seeing hex 86 - daggers

And in other columns later seeing hex 18 - CANCEL - and in the same record later, other types of non-printable chars..

I checked the source file in UltraEdit outside of DataStage and see the same thing. I've also been told we pull the file via FTP in BIN mode. So Question is, how do I:

A: Get the MF system providing the file to stop inserting non-printable chars in the fields they currently are when no data is provided in them? I say that because in the DataStage debugger I see empty fields.

B: If they persist, how can I configure the CFF stage or in the DataStage job, eliminate these non-printable chars?

I also stumbled on the following technote, albeit we are on 11.5 and not 8.7:

http://www-01.ibm.com/support/docview.w ... wg21613309

Is it possible on the MF the customer has set unicode for character fields? I wasn't aware the MF was unicode aware/capable. Oh and I tried the peek stage trick and still see the same non-printable chars.

TIA for any insight or revelations.

ff
The UNIVERSE is vast and expansive
I never Metadata I didn't like
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

Some preliminary questions:

What is the actual copybook definition for the record? I ask because "DISPLAY_NUMERIC" is a subset of alphanumeric, just restricting the valid values to numeric characters. It will have explicit bytes for the sign, commas and decimal, and sometimes DataStage will "choke" on that. For example:

Code: Select all

PIC +99,999.99 is not Decimal(7,2), it's Char(10).
Have you tried using an alternate table definition for the input? If the expected record length is (again, for example) 100 bytes, try reading the file with a single column of Char(100). If you get an import error, that will point to the actual table definition being a bad translation from the copybook.

Non-alpha characters in a character field is caused by the program that writes the record. The upstream input might be at fault as well, but you need to see how the record is being written, and whether the copybook for the write is the same one you're using for the read.

Does the copybook have any redefines sections? You might think you're reading a character field when it's filled with binary or packed numeric data.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
ffsw13
Premium Member
Premium Member
Posts: 31
Joined: Mon Feb 01, 2016 10:29 am
Location: USA
Contact:

Post by ffsw13 »

Thanks Franklin, let me chase down the CFD, will also take your alternate table def comments into account. Be Back.
The UNIVERSE is vast and expansive
I never Metadata I didn't like
ffsw13
Premium Member
Premium Member
Posts: 31
Joined: Mon Feb 01, 2016 10:29 am
Location: USA
Contact:

Post by ffsw13 »

Turned out we had the wrong copybook, topic closed.
The UNIVERSE is vast and expansive
I never Metadata I didn't like
Post Reply