Greetings,
I've been saddled with a group of DataStage EE and Server jobs that are suffering of the same aliment; seeing special or non-printable characters in various fields in the source records, and can't find a way to eradicate them once in DataStage. I've read the various posts in this forum on the topic but can't say I came away with a clear answer, and not being a recent Mainframe guy (programmed in COBOL a few decades ago on non-MF systems), not sure what path to go down.
Anyway, let me get on with it. to illustrate an example of the problem I'm having, I have the following sample copybook layout:
7 columns - level 2 - DISPLAY_NUMERIC
1 column - level 2 - GROUP
3 columns - level 3 - DISPLAY_NUMERIC
1 column - level 2 - GROUP
2 columns - level 3 - DISPLAY_NUMERIC
1 column - level 2 - DECIMAL - length 5
5 columns - level 2 - CHARACTER
1 column - level 2 - DECIMAL - length 7 Scale 4
4 columns - level 2 - CHARACTER
problem col - level 2 - CHARACTER - length 12 - seeing hex 86 - daggers
And in other columns later seeing hex 18 - CANCEL - and in the same record later, other types of non-printable chars..
I checked the source file in UltraEdit outside of DataStage and see the same thing. I've also been told we pull the file via FTP in BIN mode. So Question is, how do I:
A: Get the MF system providing the file to stop inserting non-printable chars in the fields they currently are when no data is provided in them? I say that because in the DataStage debugger I see empty fields.
B: If they persist, how can I configure the CFF stage or in the DataStage job, eliminate these non-printable chars?
I also stumbled on the following technote, albeit we are on 11.5 and not 8.7:
http://www-01.ibm.com/support/docview.w ... wg21613309
Is it possible on the MF the customer has set unicode for character fields? I wasn't aware the MF was unicode aware/capable. Oh and I tried the peek stage trick and still see the same non-printable chars.
TIA for any insight or revelations.
ff
CFF Stage and special or non-printable characters (again)
Moderators: chulett, rschirm, roy
CFF Stage and special or non-printable characters (again)
The UNIVERSE is vast and expansive
I never Metadata I didn't like
I never Metadata I didn't like
Some preliminary questions:
What is the actual copybook definition for the record? I ask because "DISPLAY_NUMERIC" is a subset of alphanumeric, just restricting the valid values to numeric characters. It will have explicit bytes for the sign, commas and decimal, and sometimes DataStage will "choke" on that. For example:
Have you tried using an alternate table definition for the input? If the expected record length is (again, for example) 100 bytes, try reading the file with a single column of Char(100). If you get an import error, that will point to the actual table definition being a bad translation from the copybook.
Non-alpha characters in a character field is caused by the program that writes the record. The upstream input might be at fault as well, but you need to see how the record is being written, and whether the copybook for the write is the same one you're using for the read.
Does the copybook have any redefines sections? You might think you're reading a character field when it's filled with binary or packed numeric data.
What is the actual copybook definition for the record? I ask because "DISPLAY_NUMERIC" is a subset of alphanumeric, just restricting the valid values to numeric characters. It will have explicit bytes for the sign, commas and decimal, and sometimes DataStage will "choke" on that. For example:
Code: Select all
PIC +99,999.99 is not Decimal(7,2), it's Char(10).
Non-alpha characters in a character field is caused by the program that writes the record. The upstream input might be at fault as well, but you need to see how the record is being written, and whether the copybook for the write is the same one you're using for the read.
Does the copybook have any redefines sections? You might think you're reading a character field when it's filled with binary or packed numeric data.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872