DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
gsbrown



Group memberships:
Premium Members

Joined: 23 Sep 2002
Posts: 140
Location: USA
Points: 1827

Post Posted: Fri Aug 25, 2017 7:58 am Reply with quote    Back to top    

DataStage® Release: 9x
Job Type: Parallel
OS: Unix
Additional info: How do we report on usage/activity?
Looking for advice and tips on how we can monitor usage of WISD DataStage jobs running as web services?

We've recently deployed a set of new services to support a 3rd party application and I'm looking to help report the service usage activity. (ie. peak usage time, low usage time, avg daily usage, etc...). There are several features of this application that use different WISD services and we're curious which ones are getting used more than others by analyzing the frequency of calls to the service and potentially identify any load stress issues.

What tools have others used or know about that could help with this? I would be looking for a capture of every single API call and the timestamp when it occurred or at the very least hourly statistics.
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2665
Location: USA
Points: 19822

Post Posted: Fri Aug 25, 2017 8:33 am Reply with quote    Back to top    

We use the concept of an audit log to log every ISD request. In the beginning of the job design, copy selected input columns and/or generated columns, such as a current system timestamp, into a Sequential File stage.

Include #DSJobInvocationId# in the file name. Set the "File Update Mode" to Append. Set "Cleanup On Failure" to False (important). That last setting defaults to True, and in cases where the job aborts or gets stopped unexpectedly, the True setting results in loss of your current audit log file. Set "First Line is Column Names" to False. Reject Mode = Continue.

Then all your audit logs accumulate over time and can be grep'ed, parsed, or read by another job and loaded somewhere for reporting if you wish.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
gsbrown



Group memberships:
Premium Members

Joined: 23 Sep 2002
Posts: 140
Location: USA
Points: 1827

Post Posted: Fri Aug 25, 2017 8:55 am Reply with quote    Back to top    

Thank you for those sequential setup tips. The idea of that had crossed my mind, but concerned about the added service overhead. I'll give that a try and see if there's any noticeable performance difference. That's going to be our best solution if there's not a good overall scraping/combing mechanism that could be plugged in from the outside.
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2665
Location: USA
Points: 19822

Post Posted: Fri Aug 25, 2017 1:54 pm Reply with quote    Back to top    

Great! I won't take credit for the idea. I must have run across it in a document at some point. It works well for our environment. I also was unable to locate any out-of-the-box logs that already contained that level of detail. I think you have to create your own. I would be curious if you are able to measure any impact from it.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1166

Points: 7706

Post Posted: Fri Aug 25, 2017 2:13 pm Reply with quote    Back to top    

9.1 can be set up to use the DSODB databse. I believe you can track usage via that and simply query the database for that job name + invocation id run stats.
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3724

Points: 29740

Post Posted: Fri Aug 25, 2017 9:16 pm Reply with quote    Back to top    

One possibility is to do it directly withjn WAS. While not exposed in the ISD GUI, ISD services, in the end, are deoloyed as standard J2EE Application Archives.....or EARs. Its been a looong time ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours