Anyway to force job to run on all compute nodes on a grid?
Moderators: chulett, rschirm, roy
Anyway to force job to run on all compute nodes on a grid?
I am suspsicious that the ulimit settings are not as they should be across all of our compute nodes in the grid. I would like to create a sequence/job that will run on each compute node in the grid and return the results of a couple ulimit statements. I have the few statements that are required (ulimit-Sa; ulimit -Ha) in a shell script.
I would like the process to be as generic as possible so I can port it to different environments (grids) and run without modification.
However, the only way I can think of to control which node a job runs on is via a config file and I don't really want to create a bunch of config files.
Am I brain dead? Is there a way to do this? Or am I reinventing a wheel here and there is some other way than datastage to do this?
TIA
Bob
I would like the process to be as generic as possible so I can port it to different environments (grids) and run without modification.
However, the only way I can think of to control which node a job runs on is via a config file and I don't really want to create a bunch of config files.
Am I brain dead? Is there a way to do this? Or am I reinventing a wheel here and there is some other way than datastage to do this?
TIA
Bob
Bob
-
- Participant
- Posts: 39
- Joined: Sun Apr 15, 2007 11:30 pm
Re: Anyway to force job to run on all compute nodes on a gri
I used peek stage to display result on each node and using ulimit -ah in job start subrutine,that should work for you.
"Books are as useful to a stupid person as a mirror is useful to a Blind person."
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
When you create configuration files for grid execution you don't specify the exact nodes, only the number that you require (at two levels). The grid management software actually allocates machines to nodes. This is fully documented in the manuals.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Thanks for the response Ray. I do understand the dynamic nature of building the config file.
What I am trying to find is a way to coax the grid management software to running my job on each of the compute nodes so that I can see the ulimit settings on each and every node.
My thought regarding creating config files would have required NOT grid enabling the job.
What I am trying to find is a way to coax the grid management software to running my job on each of the compute nodes so that I can see the ulimit settings on each and every node.
My thought regarding creating config files would have required NOT grid enabling the job.
Bob
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
OK, an explicit configuration file will do it. Create a job that uses an External Source stage to execute hostname ; ulimit -a and capture all the lines of output into wherever makes sense. This operator should execute on each compute node.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Actually it does not matter to me if it is a grid enabled job or not. As long as I can get the job to run on all the nodes in the grid.lstsaur wrote:So you mean that you want to run a non-grid job on every compute node in a grid?
All I am trying to do is confirm the ulimit settings on each server.
Bob
Now we are getting to the heart of my question. I have the job that captures the ulimit output. But, how do I get it to run on all the compute nodes?ray.wurlod wrote:OK, an explicit configuration file will do it. Create a job that uses an External Source stage to execute hostname ; ulimit -a and capture all the lines of output into wherever makes sense. This operator should execute on each compute node.
Is the only way to put one job in a sequence for each compute passing an explicit config file for each job?
Bob
Well, 2 reasons basically:
1 - If you are referring to a Unix Admin, because they have no way to see what the vaule of ulimit is as seen from a DataStage job. It is often different from what is seen by just issuing commands on the unix command line.
and
2 - If you are referring to a DataStage Admin, because I am the admin however unfortunate that might be.
1 - If you are referring to a Unix Admin, because they have no way to see what the vaule of ulimit is as seen from a DataStage job. It is often different from what is seen by just issuing commands on the unix command line.
and
2 - If you are referring to a DataStage Admin, because I am the admin however unfortunate that might be.
Bob
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
That, or a job that has a configuration file mentioning every compute node.bobyon wrote:Is the only way to put one job in a sequence for each compute passing an explicit config file for each job?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.