DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
samyamkrishna



Group memberships:
Premium Members

Joined: 04 Jul 2006
Posts: 256
Location: Toronto
Points: 1577

Post Posted: Thu Sep 14, 2017 8:30 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Additional info: Reading HDFS files
Hi,

I am able to read the data from HDFS files.
the folder structure as below.

/data/projectname/zonename/dbname/tablename/partfilexxxxx*

Question:

How do I read the read if the data is stored in partition on business_effective_date like below

/data/projectname/zonename/dbname/tablename/effective_date=20170915/partfilexxxxx*
/data/projectname/zonename/dbname/tablename/effective_date=20170916/partfilexxxxx*

Should I read them separately or is there a way to read from all the effective_date sub folders at once using file connector stage?

_________________
Cheers,
Samyam
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54006
Location: Sydney, Australia
Points: 293010

Post Posted: Fri Sep 22, 2017 3:44 am Reply with quote    Back to top    

That should all be handled automatically for you. You (the user) should remain unaware of how Hadoop partitions its data.

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours