DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
Pranita
Participant



Joined: 19 Jul 2007
Posts: 3

Points: 39

Post Posted: Wed Feb 27, 2008 5:59 am Reply with quote    Back to top    

DataStage® Release: 7x
Job Type: Parallel
OS: Unix
Additional info: Parallel execution problem
Background
There is a limit to the number of jobs that can run on ControlM (ControlM is a scheduling tool that invokes the DS jobs through scripts.) server, Due to this we are trying to reduce the no. of ControlM jobs. There are 220 jobs in the application. Creating 220 scripts (one script per job) is not feasible as these jobs will run for 200+ site in which case we will have approx. 200*220=44000 ControlM jobs on ControlM, which exceeds the ControlM limit.

Problem Description

We have created a generic script, that invokes the job passed to the script as a parameter. The parameter required for the job are identified and assigned values at run time.

The plan is use a sequence job that will call this generic script once for each job in the sequence, for invoking the script we have used the execute command activity, but the problem with this is the execute command activity runs sequentially even though these stages are placed in parallel in sequencer job.

One work around this problem is to call a dummy job from a job activity in the sequence job and inturn this dummy job invokes the generic script. Since the job activity stage runs in parallel it indirectly executes the scripts in parallel. But again here the problem here is the director log for this dummy job would be huge as the same dummy job would be called for 220 jobs in the appliaction multiplied by the number of sites (220*200)

Note:- We cant call the jobs directly from job acctivities since there as some scripts which needs to be called after execution of each job and also we cannot call these scripts in "after job subroutines" as its against the best practices defined for project.

Please let me know if you have any solutions.
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54407
Location: Sydney, Australia
Points: 295092

Post Posted: Wed Feb 27, 2008 6:10 am Reply with quote    Back to top    

What is it that makes such restrictions "best" practices? Who said?

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
balajisr


since October 2006

Group memberships:
Premium Members

Joined: 28 Jul 2005
Posts: 785

Points: 3770

Post Posted: Wed Feb 27, 2008 6:12 am Reply with quote    Back to top    

Quote:
We cant call the jobs directly from job acctivities since there as some scripts which needs to be called after execution of each job and also we cannot call these scripts in "after job subroutines" as its against the best practices defined for project.



If i understand your problem correctly, Can't you call after job script like this in job sequence. Confused

Code:
 Job Activity -------OK Trigger ------->Exec Command
Rate this response:  
Not yet rated
Pranita
Participant



Joined: 19 Jul 2007
Posts: 3

Points: 39

Post Posted: Wed Feb 27, 2008 7:41 am Reply with quote    Back to top    

ray.wurlod wrote:
What is it that makes such restrictions "best" practices? Who said? ...


Thanks for your comments.

By best practices, I meant that, in my project we are advised against the use of After-job subroutines.


balajisr - We tried using the Execute command activity after the job activity. The problem here again is that, the job activities will run in parallel, but the exec command activities will still run sequentially even though designed for parallel execution
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54407
Location: Sydney, Australia
Points: 295092

Post Posted: Wed Feb 27, 2008 4:51 pm Reply with quote    Back to top    

WHO advised and, more importantly, for what reason? After-job subroutines are a very useful part of the product. I put an annotation on the canvas to alert future developers to the use of before ...

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
Pranita
Participant



Joined: 19 Jul 2007
Posts: 3

Points: 39

Post Posted: Fri Feb 29, 2008 12:20 am Reply with quote    Back to top    

ray.wurlod wrote:
WHO advised and, more importantly, for what reason?

After-job subroutines are a very useful part of the product. I put an annotation on the canvas to alert future developers to the use of before/a ...


Hi,

I would like to reframe my query to "How can one run EXECUTE COMMAND activities parallely in a sequence job? We've noticed that although we put them in parallel, they actually run sequentially"
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42796
Location: Denver, CO
Points: 220596

Post Posted: Fri Feb 29, 2008 8:22 am Reply with quote    Back to top    

Can you post (or draw) an example where you have put the Execute Command stages 'in parallel'?

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54407
Location: Sydney, Australia
Points: 295092

Post Posted: Fri Feb 29, 2008 3:36 pm Reply with quote    Back to top    

Job sequences only ever run sequentially, and only ever on the DataStage server machine itself, not on any other node in an MPP or grid environment. That's because a job sequence is really a spec ...

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
atulgoel
Participant



Joined: 03 Feb 2009
Posts: 81
Location: Bangalore, India
Points: 866

Post Posted: Wed Aug 08, 2018 8:34 am Reply with quote    Back to top    

Hi Pranita, Do you find the solution for your problem. I am also having the same issue.

_________________
Atul
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42796
Location: Denver, CO
Points: 220596

Post Posted: Wed Aug 08, 2018 11:22 am Reply with quote    Back to top    

Rather than ask someone who hasn't posted here in ten years it would be better to start a new topic with the gory details of your version of this issue and we'll go from there.

Thanks.

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours