Consumed more than 100000 bytes looking for record delimiter

[AntRan]

I know this topic has been approached many times and the answer is always "datastage couldnt find the record delimiter".

However, in my case for testing purposes, I am building a file on aix that goes from 31990 to 32010 in length incrementing in length by 1 each time.

I run the job after I build the file. Btw, it only has the letter 'A' in it repeated that many times to make the length of the file.

When i get to record length 32001 (32000 A's plus one byte for end of line marker) , Datastage abends with the message:
Consumed more than 100000 bytes looking for record delimiter; aborting

Here is the file i am building and using to read:
wc -l temp.out
1 temp.out

ls -altr temp.out
-rw-r--r-- 1 dsxxx dstage 32001 Sep 19 09:58 temp.out

Obviously, the problem isnt the setting of the record delimiter since it is the same from file lengths 31990 to 31999 and is the same for 32000.

I read the one record file into one variable in the datastage sequential file reader that is set to LongVarChar 45000.

There have been posts about adding APT values but the document states that they are set to 100k or more so I dont see how that is a problem. when I run the job, those variables are not listed so I am not sure how to show what their value truly is.

Obviously, when it dies at 32001, one has to think there is a 32k limit set somewhere .. but I am not sure how to figure it out. One might suspect datastage would indicate it better rather than saying it is reading 100000 bytes (when there isnt that much data) to find the record delimiter.

Also, if it is really the $APT_MAX_DELIMITED_READ_SIZE, I can not find the document on how to set that correctly. I tried to add it to thru edit -> job properties -> parameters -> add environment variable. This added it to the user defined list as 100000 and either it didnt work or that wasnt the problem because i still have the error OR i added the parameter in the wrong section. I would have thought it should have been a REAL environment variable rather than one i create.

Also, i read in posts that datastage doesnt have limits on record length .. somehow with all these variables to set, there must be some limit.

[chulett]

Which 8.x release are you running?

[AntRan]

Which 8.x release are you running?

8.5

[Madhumitha_Raghunathan]

Hi AntRan,

We also faced the same issue "Consumed more than 100000 bytes looking for record delimiter" when we were rocessing large xmls in a single field.

And we got it resolved by setting $APT_MAX_DELIMITED_READ_SIZE as suggested by IBM when we raised a PMR. This variable is generally not available in the list of variables in the Administrator.

We did the same thing that you did, added the variable in the user defined section with the "$" symbol in front. We didnt set any default value in the Admin. We added this parameter to that job that was failing and set it to a value of 400000. Then the job ran fine.

[Madhumitha_Raghunathan]

The reason we need to increase the value is that it is 100000 by default and if we set the APT_MAX_DELIMITED_READ_SIZE variable to 100000 again it wont make a difference.
If you increase the value to 400000 or so it will read string lengths upto the byte length you specify.

[AntRan]

Holy Smokes .. by making that 400000, I got the file to be read.
Who knew that a 32000 btye file would not work with 100k btye read size value.

Thank you very much .. I should have tried to make the value larger to start with but the 100000 seemed large enough.

[Madhumitha_Raghunathan]

I think it is 32000 characters and 100000 bytes. When it moved to the 32001st character then it broke.

[Kryt0n]

That would require 4 byte characters, which the OP wasn't using. Not that I have a good reason as to why 400K should work

[ulab]

Hi Thought of updateing with the resolved URL which I got for the same issue, it will be usefull for others:

viewtopic.php?p=383814