DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
joycerecacho
Participant



Joined: 26 Aug 2008
Posts: 292

Points: 5411

Post Posted: Wed Feb 27, 2019 12:17 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hi everybody.

I am trying to read a txtfile from HDFS using the object "File Connector".
The file has about 720 millions of registers, 51 columns - the majority is varchar(200) and I can´t read it without the following error:

"HDFS,0: com.ascential.e2.common.CC_Exception:
An exception occurred: com.ascential.e2.common.CC_Exception Failed to parse row 3,322,164
(com.ascential.e2.common.CC_Exception:
The file contained insufficient data for column COD_ORGM.)".


The column COD_ORGM is the first one.

I know that reading the error message we suppose the problem is in the file. I read a post which says that you should check the fulfillment of the last column or, they ask you to increase the char column size to 250 (dont know why).
In short: it also happened with other files, smaller ones, and the unique thing that I did was restart them. And it perfectly worked.
So it makes me believe that the problem is not with the file.

In this case when I restart it, the error occurs in different moments: sometimes after reading 270 millions, sometimes after 3 millions.
I can´t understand.
I raised and reduced the Yarn Container Size through Datastage environment variable and it does not make any difference.

I do need to deliver these data soon.
Does anybody have any tip, please?

Best regards,

_________________
Joyce A. Recacho
Săo Paulo/SP
Brazil
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours