Grid APT configuration file

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
Andreas B
Participant
Posts: 3
Joined: Thu Mar 29, 2018 6:54 am
Location: Germany

Grid APT configuration file

Post by Andreas B »

Hi, I am currently working on a job configuration monitoring in an GRID environment. We would like to know what compute nodes our LSF decided to use. And then why, but thats another story.

I have some trouble to find the correct filename of the APT configuration file reliably. When the GRID has generated the APT configuration file for a job execution, how will it tell the job which file to use? Will the file name always be set as a parameter $APT_CONFIG_FILE? Is there a special pattern in the filename to recognize the correct file?

Of course I could get all the logs of the job run and parse the file name out of it. But for that I would have to get the last wave number first...

Maybe you have another idea?
Performance Monitoring - Speedgain For DataStage (https://www.itgain-is.de/en/performance ... datastage/)
skathaitrooney
Participant
Posts: 103
Joined: Tue Jan 06, 2015 4:30 am

Post by skathaitrooney »

If LSF dynamically generates the config file (APT_GRID_ENABLE=YES) then APT_CONFIG_FILE variable is of no use. It is only used when config files are not dynamically generated.

To find out which config file your job uses, check the job logs. Within first 10 lines of the logs you would find the name of the config file along with the path, look closely.

There is a pdf from IBM that explains how resource managers such as LSF are supposed to work with datastage
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

Almost correct.


LSF is not he one to generate the config file. The Grid Enablement Toolkit is.

LSF simply gives you a hostname or two. There is a SMALL "sleep 5" script that then poll for your main job to be finished. The sleeper program is the only job truly submitted to the grid. The DataStage application does not use LSF to actually submit the datastage job, it just used it to get a hostname, put a looping sleeper job there (to reserve your presence on the grid) and terminate it when the main datastage job is done.


But... the GRID ENABLEMENT TOOLKIT is the one that takes the hostname and makes an APT file out of it based upon your JOBDIR path in the global_grid_values file. The naming convention is based upon jobname, invocation id, time, pid, and some pixie dust.

APT_GRID_CONFIG is the value used for the config file TEMPLATE.
The toolkit done not superimpose or modify APT_CONFIG_FILE.


Look in your $GRIDHOME for the various scripts.


Andreas, can you define a bit more of what you mean by configuration monitoring?

You CAN obtain the host information from some LSF commands.

"bjobs -l <jobid>" will give you information about which hosts were reserved for the datastage job. The jobid is the LSF job number, means nothing to datastage, but it is echoed in the job log.


If you explain what you are looking for, we can come up with a good way to track it.

But ya... grid jobs have nothing to do with APT_CONFIG_FILE.
Andreas B
Participant
Posts: 3
Joined: Thu Mar 29, 2018 6:54 am
Location: Germany

Post by Andreas B »

Thanks skathaitrooney. I am trying to avoid using the logs so I am looking for another way to get the information.

PaulVL. Basically I am looking for the Information in the apt config file like nodename, hostname, disc and scratchdisc path. I already have the nodename and the hostname from the sectionleader process. So I am just looking for the disk and scratchdisk. I could simply use the grid apt config TEMPLATE in APT_GRID_CONFIG, then I know the disk and scratchdisk paths. Because its always the same for all computenodes, right? Thanks for pointing that out to me.

Configuration monitoring is a collection of prerun information like oshscript, parameter and config files for each Datastage job execution. All the information will be stored and will possibly be used to get insights about the job to find performance issues.
Performance Monitoring - Speedgain For DataStage (https://www.itgain-is.de/en/performance ... datastage/)
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

Or you could add the GRID Log message types into the DSODB database by editing CaptureLog= in your DSODBConfig.cfg file.

Then just run a SQl command to get that info... and the CPU consumed by the job.. and the Duration in seconds..

One stop shop.
skathaitrooney
Participant
Posts: 103
Joined: Tue Jan 06, 2015 4:30 am

Post by skathaitrooney »

And that is why they call Paul the DS Guru :)
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

It's because I pay them to say that.

Marketing baby!
Andreas B
Participant
Posts: 3
Joined: Thu Mar 29, 2018 6:54 am
Location: Germany

Post by Andreas B »

Well that is also a nice solution! I'll try that out for sure.

Thanks!
Performance Monitoring - Speedgain For DataStage (https://www.itgain-is.de/en/performance ... datastage/)
Post Reply