Better logging for specified metrics
Posted: Fri Oct 04, 2013 2:34 pm
DataStage 8.5
Looking for a way to enable logging of specific metrics for any given production DataStage job.
This is in effort to monitor performance on reoccuring jobs as they come up with various concerns.
Specifically, is there an automated notification to generate the below info for each run of a DataStage job.
1. # of inserts processed during that run
2. # of updates processed during that run
One option is:
Of course Director logs all this information.
All the needed stats are in there...however, it's very tedious to compile several runs worth of statistics. They are 'buried' in other notifications (filtering doesn't help much), separated by nodes (we have 12, which makes it tedious to pull the needed stat numbers from the end of a bunch of text..12 times, for each run). And then of course Monitor only shows last run.
Second option is:
To modify the DataStage job to create a branching flow that aggregates the needed statistic and updates a custom log file or table.
But we would have to create this for every metric we are looking for on that job. We don't want to have to modify each production job in this way if we don't have to.
Third option is:
To enable auditing at the database level per table.
However it would be nice if DataStage could spit that out since any particular job could touch many tables. Similiar concern as with the second option.
So we are basically wondering if there is another way to enable specific logging/auditing through DataStage that could be applied to any existing job 'on the fly', that we haven't explored. If I could get Director to filter and aggregate this information better, this might do it for us.
Thanks so much.
Looking for a way to enable logging of specific metrics for any given production DataStage job.
This is in effort to monitor performance on reoccuring jobs as they come up with various concerns.
Specifically, is there an automated notification to generate the below info for each run of a DataStage job.
1. # of inserts processed during that run
2. # of updates processed during that run
One option is:
Of course Director logs all this information.
All the needed stats are in there...however, it's very tedious to compile several runs worth of statistics. They are 'buried' in other notifications (filtering doesn't help much), separated by nodes (we have 12, which makes it tedious to pull the needed stat numbers from the end of a bunch of text..12 times, for each run). And then of course Monitor only shows last run.
Second option is:
To modify the DataStage job to create a branching flow that aggregates the needed statistic and updates a custom log file or table.
But we would have to create this for every metric we are looking for on that job. We don't want to have to modify each production job in this way if we don't have to.
Third option is:
To enable auditing at the database level per table.
However it would be nice if DataStage could spit that out since any particular job could touch many tables. Similiar concern as with the second option.
So we are basically wondering if there is another way to enable specific logging/auditing through DataStage that could be applied to any existing job 'on the fly', that we haven't explored. If I could get Director to filter and aggregate this information better, this might do it for us.
Thanks so much.