Better logging for specified metrics

SUMAN CHILUVURI · Post by **SUMAN CHILUVURI** » Fri Oct 04, 2013 2:34 pm

DataStage 8.5

Looking for a way to enable logging of specific metrics for any given production DataStage job.
This is in effort to monitor performance on reoccuring jobs as they come up with various concerns.

Specifically, is there an automated notification to generate the below info for each run of a DataStage job.
1. # of inserts processed during that run
2. # of updates processed during that run

One option is:
Of course Director logs all this information.
All the needed stats are in there...however, it's very tedious to compile several runs worth of statistics. They are 'buried' in other notifications (filtering doesn't help much), separated by nodes (we have 12, which makes it tedious to pull the needed stat numbers from the end of a bunch of text..12 times, for each run). And then of course Monitor only shows last run.

Second option is:
To modify the DataStage job to create a branching flow that aggregates the needed statistic and updates a custom log file or table.
But we would have to create this for every metric we are looking for on that job. We don't want to have to modify each production job in this way if we don't have to.

Third option is:
To enable auditing at the database level per table.
However it would be nice if DataStage could spit that out since any particular job could touch many tables. Similiar concern as with the second option.

So we are basically wondering if there is another way to enable specific logging/auditing through DataStage that could be applied to any existing job 'on the fly', that we haven't explored. If I could get Director to filter and aggregate this information better, this might do it for us.

Thanks so much.

ArndW · Post by **ArndW** » Sat Oct 05, 2013 5:49 am

Another option is to use the DataStage API calls from BASIC to get the stage and link information. Note that if a database stage does an "upsert" you won't see any information on how many of those were inserts or updates.

chulett · Post by **chulett** » Sat Oct 05, 2013 7:17 am

Another option - upgrade to at least the 8.7 release and take advantage of the Operations Console which tracks and reports data of that nature.

ray.wurlod · Post by **ray.wurlod** » Sat Oct 05, 2013 3:55 pm

Download ETLstats (for free) from Kim Duke's website - much of what you want is there and you can adapt it, since source code is included.

For your second option consider a generic shared container.