DSXchange

Hi All,

Is there a way to list how much Scratch space is used/allocated by each job?, including shared containers. I need to identify job for possible redesign to use fewer resources.

Thanks in advance for any help.

Something like this perhaps?

Edited to add: Not something I ever had an opportunity to use, just something I know is there and which gathers information of that sort. Hopefully of exactly that sort.

Scratchdisk is there for when you exhaust allocatable memory. One way to reduce demand for scratchdisk, therefore, is to add memory to your machine. Another is to specify greater than default for such things as the Sort stage, the tsort operator, buffer sizes, and so on.

Usually these kind of things are monitored in 'real time' and will vary based on data volumes. Then of course there are considerations for what other jobs are running at the same time and what other processes (O/S or otherwise) are going on as well consuming memory and disk resources. IBM has a Performance Monitoring topic online which includes tools that may help you with both the as-they-run and post-mortem approaches. Looks like the Performance Data topic may be more like what you seek.

And as noted, jobs may not be some much 'poorly written' as poorly... configured.

Files on scratchdisk should be deleted when the process that created them is finishing. Therefore there is no "post mortem" record of how much was used.

If a person wanted to track the scratch disk usage... they could set up a before and after job script to basically do a "df /themount >>/path/jobname_themount.log"; sleep 30; repeat. Make it a background task with a file watcher to terminate the job if that file is present. In your after job script, do a touch on that file.

That way you would have a background script running while that particular job is running. You would dump the size of the mount every 30 seconds to a log file.

I would absolutely NOT have this running as a permanent fixture to your ETL flow. It would only be done for stats gathering once in awhile.


As mentioned by everyone else, there is nothing built into the DataStage application that monitors the ongoing size of your scratch disk (or any other mount).

Please keep in mind that multiple jobs running at the same time, using the same APT file will write jobs to the same scratch disk mount. So your size calculation is not just based upon the ONE job you think you are monitoring. You are monitoring the MOUNT at the time the job runs. Other folks may be writing to it at the same time.

It's also highly dependent upon how you created your scratch mount as well. Is it a unique mount or simply a subdirectory on an existing mount that is also subject to read/write activity by others?

If it is used by others, then you might need to use a du command rather than df, but the du command would take more time to execute.

Also... you need to understand if the job was split across multiple hosts since the APT file allows for that. You would have to ensure that your script fetches the size of the scratch disk across all possible "fastnames" in your apt file.

Not impossible, just needs to be planned out.