Node.apt file on Virtual server

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
tine_bi
Participant
Posts: 18
Joined: Mon Nov 24, 2014 6:02 am

Node.apt file on Virtual server

Post by tine_bi »

Hi

We currently have a server setup with 2 cpu for our prod enviroment.
We now look at virtual server setup with
2 sockets and 32 virtual processors.
Drives: E, F

Our current node.apt

Code: Select all

{
	node "node1"
	{
		fastname "DSsrv1"
		pools ""
		resource disk "E:/Datasets" {pools ""}
		resource scratchdisk "F:/Scratch" {pools "sort"}
		resource scratchdisk "E:/Scratch" {pools "sort"}
		resource scratchdisk "F:/Buffer" {pools "buffer"}
		resource scratchdisk "E:/Buffer" {pools "buffer"}
	}
	node "node2"
	{
		fastname "DSsrv1"
		pools ""
		resource disk "E:/Datasets" {pools ""}
		resource scratchdisk "E:/Scratch" {pools "sort"}
		resource scratchdisk "F:/Scratch" {pools "sort"}
		resource scratchdisk "E:/Buffer" {pools "buffer"}
		resource scratchdisk "F:/Buffer" {pools "buffer"}

	}
}
This works under current settings. But should we change this in a virtual setting, and if so to what?

Appreciate any input as this is unknown terrain for me.

BR
Dan
BR
Dan
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

Don't forget to add pool "" to your scratchdisk entries.

Your quantity of logical nodes does not need to match the quantity of physical CPUs (or virtual CPUs).

You could have your node.apt file with 8, 16, 32, etc... entries, not just 2.

Performance would take a hit the higher you go, but you COULD do it.

You have to balance degrees of parallelism (your 2 nodes) multiplied by the quantity of concurrent jobs, factoring in the complexity of the jobs, in order to get a sense of how much you will stress your itty bitty PROD box.

Remember that your virtual box sounds bigger than your physical, you may have to see how that affects your licensing.
tine_bi
Participant
Posts: 18
Joined: Mon Nov 24, 2014 6:02 am

Post by tine_bi »

PaulVL wrote:Don't forget to add pool "" to your scratchdisk entries.
So I should have it like this then

resource scratchdisk "E:/Scratch" {pools "" "sort"}

After reading several posts and articles I now will test having 4 entries not just 2.
BR
Dan
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

Are you having issues with the duration of your job?
tine_bi
Participant
Posts: 18
Joined: Mon Nov 24, 2014 6:02 am

Post by tine_bi »

Yes, we experience that we now need to expand the node.apt file to have 4 nodes, not just two.

And I now want to understand why :)


Dan
BR
Dan
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

I would argue that, in a Development environment, you only ever need a one-node and a two-node configuration file. The only exception is where you need to develop with node pools. If a job runs on two nodes, it will run on 2000.

Virtual server makes no difference.
Switching to a cluster or grid topology would make a difference.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply