DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
tbtcust



Group memberships:
Premium Members

Joined: 04 Mar 2008
Posts: 220

Points: 1793

Post Posted: Sun Jun 30, 2019 12:58 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hi All,

Is there a way to list how much Scratch space is used/allocated by each job?, including shared containers. I need to identify job for possible redesign to use fewer resources.

Thanks in advance for any help.
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43026
Location: Denver, CO
Points: 222086

Post Posted: Mon Jul 01, 2019 12:09 am Reply with quote    Back to top    

Something like this perhaps?

Edited to add: Not something I ever had an opportunity to use, just something I know is there and which gathers information of that sort. Hopefully of exactly that sort. Wink

_________________
-craig

Peaches come from a can, they were put there by a man
If I had my little way, I'd eat peaches every day
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54537
Location: Sydney, Australia
Points: 295725

Post Posted: Wed Jul 03, 2019 10:26 pm Reply with quote    Back to top    

Scratchdisk is there for when you exhaust allocatable memory. One way to reduce demand for scratchdisk, therefore, is to add memory to your machine. Another is to specify greater than default for such ...

_________________
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Rate this response:  
Not yet rated
tbtcust



Group memberships:
Premium Members

Joined: 04 Mar 2008
Posts: 220

Points: 1793

Post Posted: Thu Jul 04, 2019 7:53 pm Reply with quote    Back to top    

Thanks chulett and Ray.

I was hoping to take a postmortem approach to the problem. Instead of estimating resource use one job at a time I’d like to identify what resource have been actually used (at the end f the prod batch) to point out poorly written job that have made it into production.
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43026
Location: Denver, CO
Points: 222086

Post Posted: Sat Jul 06, 2019 10:01 am Reply with quote    Back to top    

Usually these kind of things are monitored in 'real time' and will vary based on data volumes. Then of course there are considerations for what other jobs are running at the same time and what other processes (O/S or otherwise) are going on as well consuming memory and disk resources. IBM has a Performance Monitoring topic online which includes tools that may help you with both the as-they-run and post-mortem approaches. Looks like the Performance Data topic may be more like what you seek.

And as noted, jobs may not be some much 'poorly written' as poorly... configured.

_________________
-craig

Peaches come from a can, they were put there by a man
If I had my little way, I'd eat peaches every day
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54537
Location: Sydney, Australia
Points: 295725

Post Posted: Sun Jul 07, 2019 11:50 pm Reply with quote    Back to top    

Files on scratchdisk should be deleted when the process that created them is finishing. Therefore there is no "post mortem" record of how much was used.

_________________
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1281

Points: 8428

Post Posted: Tue Jul 09, 2019 9:47 am Reply with quote    Back to top    

If a person wanted to track the scratch disk usage... they could set up a before and after job script to basically do a "df /themount >>/path/jobname_themount.log"; sleep 30; repeat. Make it a background task with a file watcher to terminate the job if that file is present. In your after job script, do a touch on that file.

That way you would have a background script running while that particular job is running. You would dump the size of the mount every 30 seconds to a log file.

I would absolutely NOT have this running as a permanent fixture to your ETL flow. It would only be done for stats gathering once in awhile.


As mentioned by everyone else, there is nothing built into the DataStage application that monitors the ongoing size of your scratch disk (or any other mount).

Please keep in mind that multiple jobs running at the same time, using the same APT file will write jobs to the same scratch disk mount. So your size calculation is not just based upon the ONE job you think you are monitoring. You are monitoring the MOUNT at the time the job runs. Other folks may be writing to it at the same time.

It's also highly dependent upon how you created your scratch mount as well. Is it a unique mount or simply a subdirectory on an existing mount that is also subject to read/write activity by others?

If it is used by others, then you might need to use a du command rather than df, but the du command would take more time to execute.

Also... you need to understand if the job was split across multiple hosts since the APT file allows for that. You would have to ensure that your script fetches the size of the scratch disk across all possible "fastnames" in your apt file.

Not impossible, just needs to be planned out.
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours