DSXchange

Background
There is a limit to the number of jobs that can run on ControlM (ControlM is a scheduling tool that invokes the DS jobs through scripts.) server, Due to this we are trying to reduce the no. of ControlM jobs. There are 220 jobs in the application. Creating 220 scripts (one script per job) is not feasible as these jobs will run for 200+ site in which case we will have approx. 200*220=44000 ControlM jobs on ControlM, which exceeds the ControlM limit.

Problem Description

We have created a generic script, that invokes the job passed to the script as a parameter. The parameter required for the job are identified and assigned values at run time.

The plan is use a sequence job that will call this generic script once for each job in the sequence, for invoking the script we have used the execute command activity, but the problem with this is the execute command activity runs sequentially even though these stages are placed in parallel in sequencer job.

One work around this problem is to call a dummy job from a job activity in the sequence job and inturn this dummy job invokes the generic script. Since the job activity stage runs in parallel it indirectly executes the scripts in parallel. But again here the problem here is the director log for this dummy job would be huge as the same dummy job would be called for 220 jobs in the appliaction multiplied by the number of sites (220*200)

Note:- We cant call the jobs directly from job acctivities since there as some scripts which needs to be called after execution of each job and also we cannot call these scripts in "after job subroutines" as its against the best practices defined for project.

Please let me know if you have any solutions.

What is it that makes such restrictions "best" practices? Who said?

We cant call the jobs directly from job acctivities since there as some scripts which needs to be called after execution of each job and also we cannot call these scripts in "after job subroutines" as its against the best practices defined for project.

If i understand your problem correctly, Can't you call after job script like this in job sequence.

ray.wurlod wrote:What is it that makes such restrictions "best" practices? Who said? ...

Thanks for your comments.

By best practices, I meant that, in my project we are advised against the use of After-job subroutines.


balajisr- We tried using the Execute command activity after the job activity. The problem here again is that, the job activities will run in parallel, but the exec command activities will still run sequentially even though designed for parallel execution

WHO advised and, more importantly, for what reason?

After-job subroutines are a very useful part of the product. I put an annotation on the canvas to alert future developers to the use of before/after subroutines, so that they (the developers) are not caught unawares when the subroutine does whatever it does.

ray.wurlod wrote:WHO advised and, more importantly, for what reason?

After-job subroutines are a very useful part of the product. I put an annotation on the canvas to alert future developers to the use of before/a ...

Hi,

I would like to reframe my query to "How can one run EXECUTE COMMAND activities parallely in a sequence job? We've noticed that although we put them in parallel, they actually run sequentially"

Can you post (or draw) an example where you have put the Execute Command stages 'in parallel'?

Job sequences only ever run sequentially, and only ever on the DataStage server machine itself, not on any other node in an MPP or grid environment.

That's because a job sequence is really a special case of a server job.

You can, of course, create parallel streams of processing in a job sequence, but that's designed-in parallelism, not automatic parallelism.

Hi Pranita, Do you find the solution for your problem. I am also having the same issue.

Rather than ask someone who hasn't posted here in ten years it would be better to start a new topic with the gory details of your version of this issue and we'll go from there.

Thanks.