Unable to start ORCHESTRATE Process

[Vinodanand]

Hi,
Its aparallel with the following job design .

Code: Select all

                   Lk1 Lk2 Lk3 ..Lkn
                   |     |    |    |      

 ORADB ------->      Lookup         -----------> Transformer ------->DataSet

I have 3 streams in the same job with close to 9 reference data sets and one look up stage.

This job runs fine but I get the following error at times :

main_program: Fatal Error: Unable to start ORCHESTRATE job: APT_PMwaitForPlayersToStart failed while waiting for players to confirm startup. This likely indicates a network problem.
Status from APT_PMpoll is 0; node name is node2


I reset the job and after I had rerun the same I got a different error :

node_node2: Fatal Error: Unable to start ORCHESTRATE process on node node2 (colby): APT_PMPlayer::APT_PMPlayer: fork() failed, Not enough space

When I reran it for the third time


repartition(2),0: Fatal Error: Unable to start ORCHESTRATE network connection on node node1(colby):READYWAIT failed: parallel {natural="/chshttp/dsoweb/group4/ncbd/clients/dell/hra/logs/NGNHraEductnLvlCd_129.ds", synthetic="input repartition(2)"}(2, 0)

Finally the job completed successfully when I ran it for the 4 time .

Its a 4 node apt and there is enough space on the resource disk and scratch disk .

Free memory was also available.

I thought splitting the job could help but after the final run I am not sure what needs to be done.

Any suggestions would be helpful.

Thanks,
Vinod

[kris]

3 streams in the same job? that means the increased number of operators.

As the number of operators increase per job, the amount of scratch memory/buffer required to allot for the job is also increased. Since each operator requires it own memory pool. Each operator shares its operation among nodes.

Unable to start ORCHESTRATE process in error message is an indication that you are running out of some resource temporarily which is causing the fork of a required process to fail.

Resource Disk and Scratch Disk. You need more of these per node.
And it is also advisable to have different mount points for disk and scratdisk so that there won't be any disk contention.

[ray.wurlod]

Add APT_STARTUP_STATUS and APT_PM_SHOW_PIDS as job parameters to learn how far into the startup your job gets. This message can indicate network problems but, given that it's player processes that are failing to start, I'd also suspect that the processing nodes may be too heavily loaded when the start request is made.

[shin0066]

Make sure no other big jobs are running at the time of your job runs, hopefully resources are shared at that point of time.

[Vinodanand]

Hi,
I did run with the two Environment variables set and the job ran fine as usual with no issues. I only have this issue at rare scenarios and I ahve a feeling that If my CPU is @ 100 % utilization at the time the player process starts this could be an issue.

May be ray can throw some light on this ....

Regards,
Vinod

[ray.wurlod]

It's simple supply-and-demand economics. If you have sufficient free resources then the demand for those resources can be satisfied. If you don't they can not.

[gnan_gun]

May be SECTION LEADER problem.

[sugun.karka]

i am getting the same error did you get the solution for it

[chulett]

Please start your own posts rather than jump on the end of one that's over three years old. We really have no idea how "same" your same error is, what version of DataStage you are running, what operating system - all of which could be different from here and vital in solving your particular problem.

So please, start a new post and let us know the gory details of what is happening to you there.