DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
satheesh_color
Participant



Joined: 16 Jun 2005
Posts: 182

Points: 2145

Post Posted: Sat Jan 11, 2020 7:11 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hi All,

We have a scenario that we have to load 500 million historical records from sql server to Azure data lake followed by 200k daily/incremental records data to capture the changed records and then we will need to load into data lake.

The catch here is we don't have timestamp columns. We are looking for your thoughts and assistance as well w.r.t DataStage jobs.


Thanks & Regards,
S.R
asorrell
Site Admin

Group memberships:
Premium Members, DSXchange Team, Inner Circle, Server to Parallel Transition Group

Joined: 04 Apr 2003
Posts: 1704
Location: Colleyville, Texas
Points: 23166

Post Posted: Wed Jan 15, 2020 10:28 am Reply with quote    Back to top    

My first thought is to request they add a timestamp column to the source.

Is your problem that you are trying to identify and update changed records?

I have the same problem with another client with a feed way larger than yours. They say they can't modify the source since its a mainframe file. They threw hardware at the problem to reprocess the entire feed (clear and reload) when it comes in.

_________________
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
Rate this response:  
Not yet rated
satheesh_color
Participant



Joined: 16 Jun 2005
Posts: 182

Points: 2145

Post Posted: Wed Jan 15, 2020 9:09 pm Reply with quote    Back to top    

Hi asorrell,

Thanks for your response. I am also on the same road. No change / source will not be modified.



Regards,
Satheesh.R
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours