DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
RRourke
Participant



Joined: 13 Mar 2003
Posts: 4
Location: Montreal
Points: 37

Post Posted: Tue Dec 06, 2011 8:43 am Reply with quote    Back to top    

DataStage® Release: 8x
Job Type: TX
OS: Unix
Additional info: TX job Partner_X12_Transmission.hp
Hi,

I have the following error message from TX:

IDR_CUSTOMER_ID = xxxxx
Starting time: Wed Nov 30 15:47:32 EST 2011 Map failed for IDR_CUSTOMER_ID = xxxxx The error code is 21 Ending time: Wed Nov 30 15:52:05 EST 2011

The pdf gives this as a description:
21* Input valid but unknown data found Enough of the input data to conform to the card definition was recognized, but there was more data at the end of the input stream. Enable the Trace Input setting, re-run the map and read the map trace file (.mtr).

I have done the trace (and it is huge) but nothing seems to be satisfy the error message. I have had problems like this with TX before but they usually occur at BUILD time, this problem only arises after that when the map is called and run.

Anyone had this before and how do your find the problem card/input?
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42860
Location: Denver, CO
Points: 220962

Post Posted: Tue Dec 06, 2011 9:53 am Reply with quote    Back to top    

Moved to the TX forum.

_________________
-craig

The Old Ones were, the Old Ones are, and the Old Ones shall be. Not in the spaces we know, but between them. They walk serene and primal, undimensioned and to us unseen.
Rate this response:  
Not yet rated
rep
Participant



Joined: 19 Jun 2007
Posts: 82
Location: New York City
Points: 1097

Post Posted: Tue Dec 06, 2011 2:07 pm Reply with quote    Back to top    

This is a very common error in WTX and is a run time error. Build errors on maps are completely different.

If you had an input file with a header record and a detail record and it worked fine, then opened the file up and put a string of XXX's at the end that the map didn't expect, it would give this error, because it found enough of the data fit the type tree definition, but eventually hit something that it didn't expect. The data you're running doesn't completely conform to the type tree definition, like you have text in a numeric field, or there were 7 detail records when the type tree has a maximum of 6.

Turing the trace file on would not satisfy any errors, it only builds the trace file when you run the map. Once you turn it on and run the map, read the trace file to figure out what is wrong in the input file. While it can be cumbersome and a bit cryptic, the trace file always shows what the error is. Open the trace file and search for the word "INVALID". Half the time you do so, it will bring you right to the field/data in error, though not 100% of the time. If the trace file is too big, cut your input file into smaller chunks and run them through the map, then check the trace file. Of course if you're going to do that, make sure you "cut" the file at a place that would give you good input data, like don't cut it in the middle of a record. If it’s a file of purchase order or something, cut it at the end of a purchase order/beginning of new one. If you're still having trouble after trying that, post the end of the trace file that shows the "INVALID" and I'll take a look.
Rate this response:  
Not yet rated
RRourke
Participant



Joined: 13 Mar 2003
Posts: 4
Location: Montreal
Points: 37

Post Posted: Wed Dec 07, 2011 7:47 am Reply with quote    Back to top    

Hi,

I was sure that I had done a scan for INVALID but to be sure I did it again - there are no rows with INVALID.

I tried looking for other key words and the only row that sticks out to me is the following:

Data at offset 166 ('|0|0|0|0|0|20116...') does not match TERMINATOR '<CR><LF>'

Could this cause the Error 21?
Rate this response:  
Not yet rated
rep
Participant



Joined: 19 Jun 2007
Posts: 82
Location: New York City
Points: 1097

Post Posted: Wed Dec 07, 2011 9:18 am Reply with quote    Back to top    

Yes, this could be the cause. It's expecting the terminator, but finding the data listed. It's not 100% that it's the problem, but it's worth looking at.

Let's say you have an input with 4 different record types, Header, detail 1, detail 2, detail 3. Each has an initiator, H, D1, D2, and D3. While the Header is required, none of the detail records are. There can be any number of any details record, i.e. record(S).

Let's say your input has the header record, one detail 1 and one detail 3 record.

HDATA
D1TEST
D3TEST1,TEST2

When the map goes to validate the data, it will first look for the header. Being there is only one allowed in the type tree, after the header is validated, it reads the next portion of data expecting the first field of Detail 1.

Being you have a a detail 1 record, it validates it. After it valides the first detail 1 record, it tries to validate the next record for detail 1 again because there can be any number of detail 1 records. In your file though, the next record is a detail3 record, so in the trace file, it will say something like;


(Level 1: Offset 29, len 0, comp 2 of 3, #2, DI 00000001:)
Data at offset 29 ('D3TEST1,TEST2<CR><LF>...') does not match INITIATOR 'D1'
of TYPE X'0005' (TestIniRecordD1 Data).

Now while it gives a message "does not match INITIATOR 'D1", just like you're getting a "doesn't match TERMINATOR", that doesn't mean it's the problem. It just means that because you can have any number of the detail 1 records, it has to make sure whether the data it is reading is a Detail 1 record.

The trace will then say;

(Level 1: Offset 29, len 0, comp 2 of 3, #2, DI 00000001:)
COMPONENT number 2 of TYPE X'0002' (TestIniFile Data):
occurrence 2 is optional and does not exist.

It will then try to valide the next record as Detail 2, but because your data doesn't have one, it will do that same thing for Detail 2, saying the record does not match initiator D2, then it will say D2 is optional and does not exist. It will finally read the next record as Detail 3 and pass validation.

The same thing goes for the does not match TERMINATOR <CR><LF>

Let's say you have the same file defintion, header, detail 1, detail 2, detail 3, BUT, you do not use an initiator, only on the header record has one. Instead, the map knows which record is which because Detail 1 only has one field, while Detail 2 has four field and Detail 3 has eight fields. Now let's say in your data you have a Header record and a Detail 2 record. Each record is comma delimited.


HtestData
Field1 of Detail 2, Field 2 of detail 2, Field 3 of detail 2, Field 4 of detail 2

When the map goes to validate, it reads the header record. This is the only record with an initiator, so it knows it's the header. Now, in your type tree, the next record is Detail one, which is not required, and it only has one field. The map will read; "Field1 of Detail 2" as if it were the first field of Detail 1. Now, it will expect to see the terminator of Detail 1, but, it's not a reading detail one record. The trace files will now give the error that you are getting;

Data at offset 166 (' Field 2 of detail 2, Field 3 of ...') does not match TERMINATOR '<CR><LF>'

Even though this looks like it can be the error, it's not, it's just saying that the data it's reading does not match the terminator of Detail 1, which it is expecting. The next thing the trace file will say is that detail 1 record is optional and does not exisit. It will then validate the same data as the Detail 2 record and pass validation.

So, while that message you are getting may be the problem, it doesn't mean it definitley is, it just means it is expecting a <CR><LF> at the data it is currently reading.
Rate this response:  
Not yet rated
RRourke
Participant



Joined: 13 Mar 2003
Posts: 4
Location: Montreal
Points: 37

Post Posted: Wed Dec 07, 2011 10:36 am Reply with quote    Back to top    

Thanks for the great explanation.

If the rows in the trace following the previously mentionned message are:

(Level 2: Offset 166, len 0, comp 1 of 1, #1, DI 0000002B:)
Data at offset 166 ('|0|0|0|0|0|20116...') does not match TERMINATOR '<CR><LF>'
of TYPE X'0524' (Row WK_REGTOLL_Usage_DTL Data).

(Level 2: Offset 0, len 0, comp 1 of 1, #1, DI 00000001:)
COMPONENT number 1 of TYPE X'0523' (USAGE_KEY_GRP WK_REGTOLL_Usage_DTL Data):
occurrence 1 is optional and does not exist.

(Level 1: Offset 0, len 0, comp 1 of 1, #1, DI 00000001:)
COMPONENT number 1 of TYPE X'0522' (DBSelect WK_REGTOLL_Usage_DTL Data):
occurrence 1 is optional and does not exist.

INPUT 48 exists (5157 bytes) but has no content.

End of Validation messages for INPUT CARD 48.

Would the exist but no content let us know there is a problem?
Rate this response:  
Not yet rated
rep
Participant



Joined: 19 Jun 2007
Posts: 82
Location: New York City
Points: 1097

Post Posted: Wed Dec 07, 2011 12:51 pm Reply with quote    Back to top    

Input card 48?!?!? My God…either someone did a wonderful job on a very complex mapping situation….or a terrible one.


Anything other then "Input 48 was valid", means there is a problem.

And now that I think about it, the "21" is a warning, not an error message. I can’t recall off hand what problems usually throw this error.


If all else fails, you may have to suck it up and go through the entire trace file. Usually the actual error will be at the top, or most likely, at the bottom. Another keyword to search for other then invalid is FAILED. Like I mentioned above, you can try to minimize the size of it by parsing the data. Like if it’s a file of 10 purchase orders, put the first five in one file and the second five in other file and running each one. If one fails and the other passes, you know the problem is in the one that gave you errors. You can try to parse it down as much as possible, even to one the one purchase order that is giving you errors, then read the trace file field by field to figure out what is wrong. I would make sure there’s no extra terminators in the data.

If you're going to use WTX and fix data errors, I would strongly suggest building a type tree like the test one I described above with the header records and detail records. Between building the type tree, creating a map and pointing the input card to that type, and building a quick comma delimited file to run against, it shouldn't take you more the 15 minutes. Turn the trace on, run the file, then open the input data, the trace file, and view the type tree in the input card of map designer. Go through the trace file field by field while looking at the type tree for every object and get a solid understanding of how it validates the data. This knowledge will be invaluable if you are going to be fixing data errors the map gets. Then remove a required field or put text in a numeric field, or put 3 detail records when the type tree has maximum of 2, then see what you get in the trace file.

This all should be more then enough information to follow through on how to debug the issue. If you go through with the suggestions and get stuck, then post back.
Rate this response:  
Not yet rated
jvmerc
Participant



Joined: 02 Dec 2003
Posts: 94

Points: 812

Post Posted: Mon Dec 12, 2011 1:21 pm Reply with quote    Back to top    

Start at the end of the file can count back 5157 bytes. the 1st byte before that point was the last one it could validate thru the end of a segment. After that it got lost.

If the file is 5157 bytes in total it could not identify the begining or end of segment/record/row.

It appears that it thinks your record should end at byte 166 with a carrage return line feed. "Data at offset 166 ('|0|0|0|0|0|20116...') does not matchTERMINATOR '<CR><LF>' "
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours