ISD Maximum Running Jobs

Dedicated to DataStage and DataStage TX editions featuring IBM<sup>®</sup> Service-Oriented Architectures.

Moderators: chulett, rschirm

Post Reply
pbttbis
Premium Member
Premium Member
Posts: 36
Joined: Thu Dec 11, 2014 3:30 am
Location: South Africa
Contact:

ISD Maximum Running Jobs

Post by pbttbis »

Hi,

If you look at operations console -> Workload Management -> Queue Management

There are "Maximum Running Jobs" for various queues. The parameter for the ISD queue is set at 100, but it is not configurable (at least in the console) like the others.

Anyone know how to change the value?

Thanks,

Shaun
PBT TBIS Consultant
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

Google is your friend.

https://www.ibm.com/support/knowledgece ... icies.html

I believe that is what you are looking for.
pbttbis
Premium Member
Premium Member
Posts: 36
Joined: Thu Dec 11, 2014 3:30 am
Location: South Africa
Contact:

Post by pbttbis »

Hi Paul,

Yeah that is for system wide, which I have set to 200.

What I need to be able to do is modify the max limit on ISD jobs under Queue Management.

Thanks,

Shaun
PBT TBIS Consultant
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Not sure why you can't edit it, but it's likely because it isn't that meaningful, provided that these are "always on" ISD Jobs (that have an ISDInput)...

Carefully consider your ISD Jobs...do testing to find out what their high water mark is regarding the performance you need relative to the number of instances that you need to set (within ISD), and then lock that value in as the minimum.

There are very few dynamics here, and you don't want any anyway...for real time Jobs, they need to "be there"...they cannot afford, in most cases, to ever be queued.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
pbttbis
Premium Member
Premium Member
Posts: 36
Joined: Thu Dec 11, 2014 3:30 am
Location: South Africa
Contact:

Post by pbttbis »

Hi Ernie,

The high water mark we estimated to be at 200-300.

The console is a little confusing that it is under the tab "Queue Management", but the parameter there that I want to change is "Maximum Running Jobs". It is currently set to 100 and when trying to change it get the response "Update or delete is not allowed for non batch queues"

Cheers,

Shaun
PBT TBIS Consultant
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

200 to 300?

are these individual services? do they all have isd input?

...or are you talking about 200 to 300 concurrent requests?

......thats a really huge number for isd always on jobs....

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
pbttbis
Premium Member
Premium Member
Posts: 36
Joined: Thu Dec 11, 2014 3:30 am
Location: South Africa
Contact:

Post by pbttbis »

Well for example now we are testing a workload of ~360 request per minute which we are attempting to handle with three applications created in Information Services Director. The applications are configured with 45, 45 and 10 instances as the min and max.

We may be able to reduce the 100 instances down a bit given more load testing but as the ~360 per minute load is only 40% of the expected workload we will need to create more than a 100 instances.

Are you saying ISD is not suited for the above?
PBT TBIS Consultant
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Just saying that seems like a lot. A single instance can often handle 200+ requests per second, even with a slow protocol like SOAP over HTTP. Of course it could depend on your tranformations, but if you have 45 instances your throughout could/should be a LOT higher. Something seems very awry here.

Your minimums are at 45? How many stages in the Job? Are these Server Jobs or EE? ....is is a huge machine or grid?

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
pbttbis
Premium Member
Premium Member
Posts: 36
Joined: Thu Dec 11, 2014 3:30 am
Location: South Africa
Contact:

Post by pbttbis »

Well the jobs are very light weight. EE but running on a single node.

The reason for creating so many instances is that even though the job is light weight there is a REST call made by a hierarchical stage in the job that takes anything from 5-12sec to respond. As we need to process a single request in under 30 seconds we decided to create so many instances so that we do not have requests queuing up behind that REST call. In this way we are submitting many parallels REST calls allowing us to come in at under 30 seconds reliably.

The machine is reasonable hefty. 8 cores 32GB of RAM. Saying that while the instances are up and handling requests the CPU idles at 5%. The only signficant resource used is memory. In the case of the 100 instances that is ~20GB.

The client is also looking at upgrading the machine to a size of 48 core and 256GB in the future which may be needed once we start beefing up the jobs with more transformations etc.

So the above is a lot of context and interesting to discuss, but the original post was asking how to change the Maximum Running Jobs for ISD. I have also asked the question to IBM via a PMR.
PBT TBIS Consultant
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Ok.. sounds very interesting. Just be careful. Start-up in such situations (with that many osh processes needing to come up) can be very problematic with that few cores, along with that many concurrent job instances and EE stages.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
pbttbis
Premium Member
Premium Member
Posts: 36
Joined: Thu Dec 11, 2014 3:30 am
Location: South Africa
Contact:

Post by pbttbis »

yeah I set the workload management to start up only 3 jobs every 30 seconds otherwise things do not go well.
PBT TBIS Consultant
pbttbis
Premium Member
Premium Member
Posts: 36
Joined: Thu Dec 11, 2014 3:30 am
Location: South Africa
Contact:

Post by pbttbis »

Confirmed with IBM that there is hard limit of 100 ISD jobs that can run concurrently per engine.

To work around this we are implementing a multi engine solution and load balancing request across them.
PBT TBIS Consultant
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Ok... let me know if you need any tips on how to deploy your single Operation across multiple DS Engines with automatic round-robin load balancing.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Post Reply