ps -ef command error

[abhijain]

Hello Experts,

We are using DS 8.7 version with AIX and database as Teradata 13.11.0.07

In my projects, we don't have any scheduler used to schedule datastage jobs. But, indeed we have datastage master sequencer using batch concept. We have database batches created and on run time, our master sequence scheduler read these batches. The batches contains rows and row type. For e.g.

RowType-------------------ETLName
-----------------------------------------------
Parallel-------------------ETL1, ETL2, ETL3 etc..
Sequential-------------------ETL4, ETL5, ETL6 etc..

The ETL master sequence scheduler checks the RowType, and perform the operations accordingly. For e.g. if RowType is Sequential, ETL5 gets executed only when ETL4 is finished i.e. in sequential order. For Sequential order of execution, I've used "dsjob -run -wait" command.

When my jobs are running in Sequential manner, I am getting expected result, i.e. next job waits for the execution until the current running jobs finish. I have put "ps -ef" commany by passing correct ETL names to check the status of execution. In case of sequential, it is running fine. No issues.

For parallel rowtype, all ETLs can run in parallel. To achieve this, I have used "dsjob -run " command to execute in parallel using loop activity in sequencer. The trigger check the rowtype value and accordingly, it executes dsjob command option to run the ETL names passed to it (using loop variable through some temporary file).

But, in case of parallel, to achieve parallelism, I've not used "-wait" option. I am simply using "dsjob -run" to start executing ETLs, and then going back to loop for running next jobs in step (i.e. RowType=Parallel)

Problem: In case of parallel, I have used "ps -ef" command in while loop (with sleep until status turn to 0) to check the status of all ETLs executed after the completion of loop activity. But, sometimes, "ps -ef" returns status=0 for the job, which are still running. Is there anyway, we can handle this scenario. I am not sure, why ps -ef gives wrong status of the job, even though job is running.

Is there anyway, we can handle this. I know, "dsjob -run" doesnt hold the session like "dsjob -run -wait" (it waits for the jobs to get finished). But, dsjob -run, still execute the ETL and as long as ETL is running, ps -ef shouldn't return status =0.

I was facing same problem for the sequential type also, but for sequential type, I applied -wait option and it fixed my issue. But, here for Parallel execution mode, I can't do the same. If I do, I loose parallelism.

This problems occurs randomly. Appreciate all suggestions.

Thanks all in advance.

[PaulVL]

First... ick.

Second, did you notice if your process is failing if you have particularly long datastage job names (perhaps due to invocation ids).

Your ps -ef is probabaly chopping your results. you might want a wider version of that. "ps -efww" On linux (please consult the man page on your version of ps). Your shell environment will also limit your command output for ps. Look at your COLUMNS value (consult your Unix flavor for the specific shell variable). But what I have seen is that the datastage engine was bounced in a shell window size... 40. (admin had a small window when he bounced). Engine thus picked up the shell settings and now datastage engine has a COLUMNS value of 40, which will mess up your ps -ef output. Dump your env just prior to your ps -ef command just to see what settings you are actually playing with.

third, ick. get a real job scheduler or upgrade to laters datastage version that does have that scheduling built in.

[ArndW]

You shouldn't be going to the "ps -ef" process level in any type of normal job control. You can get the status of a given job either through the DataStage API DSGetJobInfo() or the command line "dsjob -jobstatus" method if you are doing UNIX scripting.

[chulett]

You could also look into what I seem to recall a Sequence job will do in a similar situation - use DSWaitForJob() but pass it a list of job handles to wait for. And it does.

However, not aware of an equivalent from the command line, you'd need to make use of a routine and DSRunJob() to initiate them if you wanted to go that route.

[abhijain]

Hello PaulVL,

Regarding your comment

Engine thus picked up the shell settings and now datastage engine has a COLUMNS value of 40, which will mess up your ps -ef output.

I don't agree as this is happening sometimes. However, regarding the size of 40, i have seen "ps -ef" returned output where the length cross 60 also.

[abhijain]

Thanks Gurus.

I have got some work around. I am going to use Audit tables instead of trusting ps -ef command.