Datastage Multiple instance job to run 'n' times using parm

[Chandrathdsx]

Hi,

I have a datastage multiple instance job with name 'JobA', it has a parameter 'ParmA'. I want to run the job number of times of the value of ParmA. Example, if the ParmA value is = 2, the JobA has to run two times with instance id 1 and instance id2. If the ParmA value is 5, The job should run 5 instances simultaneously. I now how to run in loop (runs sequential) or run manually using invocation id 1 to 5. But I want to automate to run the number of times simultaneously depending on the passed parameter ParmA value.

Any help with this on how to implement this use Datastage Sequence?.
The challenge I am having is the value of ParmA is dynamic and it can be 2 or 5 or 10, based on this value those many instances of JobA should run simultaneously.

Thank you!

[kris007]

You will have to put JobA within another Job Sequence and then use the looping logic with Counter value set to the ParamA value. That way, JobA will run as many times you want with the correct instance ID's.

Hope that helps.

[chulett]

How will the number of times the job needs to run be determined? Sounds like 'ParamA' needs to be in the Sequence but wondering what 'fills it out'.

[Sreenivasulu]

Using loop is 'sequential' but using multi-instance is 'concurrency'. You can achieve this using loop.

Regards
Sreeni

[ray.wurlod]

Using a loop will be sequential. You need to design a separate sequence for each value of N (number of parallel iterations) and a sequence to determine N and select the correct one of these.

[Sainath.Srinivasan]

You need to use the parameter at the sequence level to determine number of times the job must run.

Use loop activity to iterate through the counter.

Within the loop, branch into 2 stages - one your job and another a sequencer (dummy placeholder).

Join both into another sequencer with 'ANY' option and link this to your end-loop.

[chulett]

Please read the original post a little more carefully.

I know how to run in loop (runs sequential) or run manually using invocation id 1 to 5. But I want to automate to run the number of times simultaneously depending on the passed parameter ParmA value.

[Sainath.Srinivasan]

Having the ANY sequencer will ensure that all instances will run simultaneously - i.e. in parallel.

Am I missing something ?

[chulett]

Yes, it would seem so. How does "iterating through a loop" get anything running simultaneously? Adding those two stages inside the loop doesn't change the basic functionality of the loop at all, each iteration still runs one job and it all happens in a sequential fashion.

[priyadarshikunal]

Its not an easy task if the ParamA is dynamic with no Max value.

In case you have a max value. Use a user variable activity and pass ParamA to a variable. Place as many Job activity as the maximum value.

In trigger define
ParamA>=1
ParamA>=2
ParamA>=3
.
.
.
ParamA>=n

Hence it will trigger as many instances a ParamA at the same time.

In case the value is too high, you can define like 10 at a time and put it in loop with sufficient conditions.


or

call a shell script which triggers a job in loop without -wait or -jobstatus specified (with invocation id as loop counter). and then checks the status in loop too every 5-10 seconds or so.

or

write your own job control.

[chulett]

Exactly, this all comes down to writing your own job control code. That's the only way to fire them off and not start to wait until all have started. The fun part of that would be keeping an array of handles in memory and checking them all properly over and over.

[ray.wurlod]

Actually that's easy, and is exactly what the job control code generated by compiling a job sequence does. In summary, it makes use of the fact that the DSWaitForJob routine can wait for an array (a dynamic array) of job handles. It returns any time any of the jobs finishes, the code can return to a wait state after that finish has been dealt with until there are no more jobs (activities) being waited for.

[chulett]

I believe I used the word 'fun' not 'hard'... it is a great learning experience. And a fun problem to solve.

Oh, and interesting factoid about DSWaitForJob(), I wonder if that has always been the case? For whatever reason and after that had-coded job control, it's something I ever realized or recall being mentioned here. Old dog, meet new trick. :D

[creatingfusion]

You will have to put JobA within another Job Sequence and then use the looping logic with Counter value set to the ParamA value. That way, JobA will run as many times you want with the correct instance ID's.

Hope that helps.

It would be better in this situation to run the job in multiple sequence using Unix Shell Scripting.....
Let the script be rundsjob.sh and the parameter be 'n' notifying the number of instances the job needs to run.
so from unix command promt you call the script as:
sh rundsjob.sh n
where n would be a numeric value you want the job to run the number of times.

now the shell script would be written as under:

#!/bin/sh
#Usage: rundsjob.sh n
i=1
while [ "$i" -le "$1" ]
do
dsjob -run <DataStage_Job_Name>.$i
#Where <DataStage_Job_Name> is the name of the DataStage job you #want to run
done
exit 0

---------------------------------------------------

Hope this solution helps....

[mctny]

So what is the resolution. I have the same requirement ; running multi instance job N times simultaneously based on the job parameter N.

Is it possible to do it with a job sequence? or it has to be a job control job?