COBOL file without copybook

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

Rumu,

Read the using mainframe FAQ linked below for details on handling packed decimal.

Your column length for PIC S9(4)V COMP-3 is 3 bytes: counting the integer places you would see in the file 01 23 4F.

As for parsing, you can define the file stage input single column as Char, even though it doesn't fit the formats of the individual columns. That will get you past your error.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
rumu
Participant
Posts: 286
Joined: Mon Jun 06, 2005 4:07 am

Post by rumu »

Thanks Franklin.

I changed the column type to Char but the job ran for more than 30 minutes and rejected all the records.

For packed decimal fields I found in data dictionary, the field length always shows 2 for PIC S9(4)V COMP-3 fields. When I define the length as 3 in the output column of column import stage, then how does DataStage identify the starting position of the next field?
Rumu
IT Consultant
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

The reference you found is incorrect.

COMP-3 storage length -- the bytes it takes up on the file -- is number of integer places divided by two and add one, round up for odd number of places.

PIC S9(4) COMP-3: 4 divided by 2, add 1, storage length is 3.
Storage, hexadecimal representation: 1,234 (or 12.34, decimal is never stored) is 01 23 4C

In the FAQ example, PIC S9(5) COMP-3 takes 4 bytes. 12,345 is 00 12 34 5C.

Please start from the beginning with the entire record, and "map" it to the copybook. Manually determine starting position and length for every field. That's the only way to be sure that your [position,length] values in the derivation are accurate.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
rumu
Participant
Posts: 286
Joined: Mon Jun 06, 2005 4:07 am

Post by rumu »

Hi Frank,

I am sharing a part of the data dictionary here: starting part

FROM TO FIELD LENGTH PICTURE
1 75 CHD-RECORD 75 X(75)
76 77 CHD-NO-SEG 2 S9(4)V COMP
78 79 CHD-ALP-SEG 2 S9(4)V COMP

when I read the abobe layout, i am using in column import stage output column tab as below:

CHD-RECORD Char 75
CHD-NO-SEG Decimal 3
CHD-ALP-SEG Decimal 3

after CHD-RECORD when I read CHD-NO-SEG will define length as 3 and packed-yes
but in case of CHD-ALP-SEG, where will be the starting position? 78 or 79 ?

I can not handle packed decimal in transformer stage as if i read the input as Char, the job rejected all the records. if I use Varbinary, char fields are read correctly but Decimal field is showing error asking to use rawto String function ,if i use that function it will mess the data.

So I switch to column import stage. Please suggest how do I define the length of the packed decimal data in column import output stage . If I go by the calculation, then will it impact the starting position of the consequitive fields?
Rumu
IT Consultant
rumu
Participant
Posts: 286
Joined: Mon Jun 06, 2005 4:07 am

Post by rumu »

You said "Please start from the beginning with the entire record, and map it to the copybook."

Shall I use a sequential file to read the entire record in one single field? Initially I was using that with a VarBinary field. While splitting the record in transformer as per the data dictionary, I faced an issue with packed decimal fields.

For Char fields I used raw to String(input_column)[start,length]

But this derivation is not applicable for packed decimal. Can you please tell me what exactly I need to use to split packed decimal fields?

My copybook is not good. For multiple record types file, the copybook has definition for only one type of record without defining the variable part within it.
Rumu
IT Consultant
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Maybe I missed that in all of this but why are you not using the Complex Flat File stage for this? :?
-craig

"You can never have too many knives" -- Logan Nine Fingers
rumu
Participant
Posts: 286
Joined: Mon Jun 06, 2005 4:07 am

Post by rumu »

Hi Craig,

The file I am reading has 11 types of records and each record type has variable length based on the segments.
The copybook that I received has layout for only one Type of records and without considering the variable length segments.
While reading the data using that copybook, I had to uncheck multiple record types as the copybook has one 01 level.
While reading the data, as expected lots of warnings received saying input buffer overrun.Ultimately 6 records are processed and with all packed decimal fields as 00000. Expected number of records are much much more.
As no body can rectify the copybook, thought of using sequential file to read the binary data as a single column and then parsing.. There also getting issues with packed decimal fields..
I am stuck ..Any help is much appreciated. Thanks.
Rumu
IT Consultant
rumu
Participant
Posts: 286
Joined: Mon Jun 06, 2005 4:07 am

Post by rumu »

Hi,

When I am trying to read the binary file in sequential stage, I read the entire row in one VarBinary column and then use RawToString function to convert and filter the specific record type from first field. I can view the Record Id in DataStage viewer and in UNIX but the string equality is not working in transformer... I tried to use Downcase function on that field and found some strange character in viewer and UNIX.

If I use the single column as Char then I am not able to read any fields... I used Conversion with MB and I used options to convert from Binary to Decimal but that did not work. Can you please help me how to read a binary file using sequential stage? VarBinary or Char? What function should be used to convert string and packed decimal?
Rumu
IT Consultant
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

When Franklin said "Map it to the copybook" I believe he meant FIX the copybook to make it reflect reality. Then use the Complex Flat File stage to read the data using the new copybook.

You will probably need assistance from a COBOL programmer on the mainframe to help you do that, but it can be done.

I have never had success loading a complex COBOL file without an actual working copybook and the CFF stage. I have had to modify the copybook a few times, but then I can do that as I was a COBOL programmer during college and afterwards. (yes - that's how old I am!)

Even if the mainframe doesn't have the copybooks anymore, there are chargeable third party tools that can parse load modules and rebuild them. I've never used one, but I have heard of customers using them before.
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
rumu
Participant
Posts: 286
Joined: Mon Jun 06, 2005 4:07 am

Post by rumu »

so without copybook, I cant read a Binary file? I saw one thread , quite old (2008) where it is mentioned to use sequential stage and column import stage ..no further details.
Rumu
IT Consultant
rumu
Participant
Posts: 286
Joined: Mon Jun 06, 2005 4:07 am

Post by rumu »

Hi,

I finally got an actual copybook of First type of record which we need to load, but the File has other 10 types of records as those are not used so copybooks not available.

I imported the copybook and configured the CFF stage.
This time the job ran with warnings but correct number of output records ie correct number of first type record is received. As the File has other 10 types of records of varying length so the warnings generated as Import warning for short records.. The last step is pending to filter out the first type record before passing to the CFF stage so that the CFF stage should process only the first type of record according to the layout.
I used a filter command in CFF stage as

grep -a '02CHK' <Filename?


02CHK is the Record ID of first type.
But this doesnot work. I tried with the Constrat tab in the output tab of CFF stage but that also does not work.

Could you kindly help me with this last thing to Filter out '02CHK' from the input Binary file before passing to CFF stage?

The input File name is MoneyMST.<datefield>[/code]
Rumu
IT Consultant
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

We would need more than "does not work" to help. Exactly what doesn't work about it? If you manually run that grep from the command line, do you get the output you expect - i.e. the proper number of lines with the values unchanged? I ask the latter because I'm not familiar with that "-a" (read binary as text) option so no clue what the ramifications of using it are.
-craig

"You can never have too many knives" -- Logan Nine Fingers
rumu
Participant
Posts: 286
Joined: Mon Jun 06, 2005 4:07 am

Post by rumu »

Hi Craig,

I used the command in the command line, it did not work as it returns 0 rows.
Rumu
IT Consultant
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

Rumu,

Please confirm that the following is accurate:

Code: Select all

FROM TO FIELD LENGTH PICTURE 
 1 75 CHD-RECORD 75 X(75) 
 76 77 CHD-NO-SEG 2 S9(4)V COMP 
 78 79 CHD-ALP-SEG 2 S9(4)V COMP 
If it is accurate, those are not packed decimal fields. They are binary numeric fields that are sql type integer or double or of that category.

COMP is not packed decimal. Only COMP-3 is packed decimal.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
rumu
Participant
Posts: 286
Joined: Mon Jun 06, 2005 4:07 am

Post by rumu »

Hi Frank,
Your are correct, the copybook given to me was wrong where all the decimal field was made as COMP-3..Yesterday I received a correct copybook where I found that the above mentioned columns are COMP hence when I imported the metadata, it shows me smallint length 4.

As I received a complete copybook of onetype of records so the last challenge remains to extract the first type of records from the binary file and pass those to CFF stage . I am now stuck with filtering one type of data from the binary file.
Could you please help on this?
Rumu
IT Consultant
Post Reply