DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
rberman



Group memberships:
Premium Members

Joined: 01 Oct 2008
Posts: 31

Points: 280

Post Posted: Tue May 19, 2020 9:16 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
I am trying to write to a Hadoop Hive csv file using a file connector. The data contains embedded linefeeds which the user would like to keep.

Has anyone done this? If so how did you define your table and what were your settings in datastage?

I am able to write to Hive files, however, the linefeed threw it for a loop.

I was advised to try to use an escape character so I tried the following which aborted. If it had not aborted I am not sure it would be correct anyway:

Using a File connecter:
File System: WebHDFS
field delimeter: ç (used because not in the data)
Row delimeter: <NL>
escape character: \
quotation mark: none

Given some guidance, i set up the hive table as follows:
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES (
'escapeChar'='\\',
'field.delim'='ç',
'line.delim'='\n',
'quoteChar'='\"', -- (I also tried running without this - so it was matching the above settings each time)
'serialization.format'=',')
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'

Error:

File_Con_ORDHED_HADOOP,0: An exception occurred: java.lang.Exception: Failed to write to file /user/hive/warehouse/salesforce_procurement.db/dev_ccm_idea__c/dev_ccm_idea__c.dat: HTTP/1.1 500 Internal Server Error
at com.ibm.iis.cc.filesystem.impl.webhdfs.WebHDFS.createFromBuffer(WebHDFS.java:279)
at com.ibm.iis.cc.filesystem.impl.webhdfs.WebHDFS.writeFromStream(WebHDFS.java:224)
at com.ibm.iis.cc.filesystem.AbstractFileSystem.writeFromStream(AbstractFileSystem.java:49)
at com.ibm.iis.cc.filesystem.FileSystem$Uploader.call(FileSystem.java:4338)
at com.ibm.iis.cc.filesystem.FileSystem$Uploader.call(FileSystem.java:4301)
at java.util.concurrent.FutureTask.run(FutureTask.java:277)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:812)
An exception occurred: java.lang.Exception: Failed to write to file /user/hive/warehouse/salesforce_procurement.db/dev_ccm_idea__c/dev_ccm_idea__c.dat: HTTP/1.1 500 Internal Server Error
at com.ibm.iis.cc.filesystem.impl.webhdfs.WebHDFS.createFromBuffer(WebHDFS.java:279)
at com.ibm.iis.cc.filesystem.impl.webhdfs.WebHDFS.writeFromStream(WebHDFS.java:224)
at com.ibm.iis.cc.filesystem.AbstractFileSystem.writeFromStream(AbstractFileSystem.java:49)
at com.ibm.iis.cc.filesystem.FileSystem$Uploader.call(FileSystem.java:4338)
at com.ibm.iis.cc.filesystem.FileSystem$Uploader.call(FileSystem.java:4301)
at java.util.concurrent.FutureTask.run(FutureTask.java:277)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:812)

Thanks in advance!
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours