DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
abyss



Group memberships:
Premium Members

Joined: 22 May 2014
Posts: 158

Points: 1655

Post Posted: Thu Nov 16, 2017 6:01 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Windows
Hi all
this topic may be a little more than only datastage

I am trying to tune a job that calculate check sum on one of our table. The table is massive, it has over 120 million rows in it and more than 60 fields.
the job call a checksum stage to calculate checksum value for each record.
the problem is this job takes too long. more than 2 hours, is there a way to tune this job make it run faster?

thanks
thompsonp



Group memberships:
Premium Members

Joined: 01 Mar 2005
Posts: 187

Points: 1709

Post Posted: Fri Nov 17, 2017 3:35 am Reply with quote    Back to top    

Can you provide some more details about the job?

What are the source and targets?
Are you reading from the database in parallel?
What else is the job doing if anything?
In the checksum what options do you have selected?
How is the job partitioned?
How large is a row in the table?

How have you determined that the checksum is the cause of the perceived poor performance?
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1166

Points: 7706

Post Posted: Fri Nov 17, 2017 8:41 am Reply with quote    Back to top    

Also, the basic question of "Why are you doing a checksum?" needs to be asked. My initial guess is some type of CDC detection...

By describing the need, we might help with a different way to obtain your end goal.
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42268
Location: Denver, CO
Points: 217033

Post Posted: Fri Nov 17, 2017 9:05 am Reply with quote    Back to top    

In addition to all of the above, I'd also be curious how many "more than 2" hours are we talking about as that doesn't really sound like all that poor of a performance for a "massive" job of that nature. To me.

_________________
-craig

Watch out where the huskies go and don't you eat that yellow snow
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours