How do you do chargeback?

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

How do you do chargeback?

Post by qt_ky »

Our organization has most of the products in the Information Server suite running on an SMP LPAR with one instance for production and another SMP/LPAR/instance for non-production. We would like to create a way to charge annual costs to individual projects, applications, and vendors. These consumers may come and go. In many cases their deliverables will continue running in production long after the people leave.

We do not have any special tools or products available for measuring actual usage. The idea is to make a shared instance available for multiple tenants (in a secure way) and to charge in a fair way. The charge could be based on actual usage, or based on an allocation, or a combination.

We are looking at as many factors as possible to consider charging for, such as infrastructure costs (LPARs, CPU, memory, disk space, backups), software administration (support, security, project setup, applying patches and upgrades, managing the infrastructure, etc.), and annual software subscription and support (licenses/maintenance).

How do you do chargeback, or how have you seen it done? What works well, what doesn't, any lessons learned?
Choose a job you love, and you will never have to work a day in your life. - Confucius
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

<crickets>

Me, I've never worked anywhere that even attempted any kind of a chargeback, so can't really help you there. But did want to at least reply. :(
-craig

"You can never have too many knives" -- Logan Nine Fingers
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

Thanks, Craig. I appreciate replies. :) Perhaps we are venturing into uncharted territory...
Choose a job you love, and you will never have to work a day in your life. - Confucius
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Due to the dearth of replies, I suspect so!
-craig

"You can never have too many knives" -- Logan Nine Fingers
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

I've never seen a good charge-back model either.

We basically adopted a "Buy a host and add it to the pool" type of charge-back. Even that is hard to justify for small potato projects. So we let them in for free'ish.

I suspect if you were going to to a CPU chargeback, you would use the DSODB console. Then mark it up a bit for overhead of WAS and DB2 hosts, department labor costs and such.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

I agree with Paul's model. However, if everyone has engine credentials mapped to the one user, you don't have the information required for defensible chargeback.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

Thanks for your replies.

Do you all think that CPU-based chargeback is the most common?

We do not do credential mapping. However, in production we schedule all production jobs under a single, non-expiring administrator ID, across all projects.

I see that the Ops Console has an Activity tab where you can select Past Activity, give it a date range, and view Resources and charts, like the CPU chart, job count, process count, and memory usage.

Maybe we do not have the granularity needed, or the console does not display it, and/or I am missing something, but I do not see any way to break it down from the totals or split anything apart across projects, etc.
Choose a job you love, and you will never have to work a day in your life. - Confucius
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

A new query based upon a "Yesterday" activity for CPU consumption would be required, not scraping the activity screen.

Perhaps if you described your political setup and how they map out to projects.

For instance, does each department have their own project and as such each job under a certain project goes to the same billable entity?
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

Are you talking about querying the operations database directly, as opposed to viewing something within the Ops Console?

Our DataStage and QualityStage projects are organized by agency. Each agency using the tools will have one or more projects associated with them. All jobs within any given project are associated with only one agency.

One of the main problems is that every agency is running for "free-ish" and our IT shop gets stuck with all the costs, and we don't have any budget.
Choose a job you love, and you will never have to work a day in your life. - Confucius
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

I was at a very large Grid customer that needed to apportion charges back to individual projects. The easiest way we found to do this was to assign a unique DataStage id to each billable group. Then we used UNIX utilities that captured usage over time on a per-id basis. Projects were then billed based on CPU / IO usage. Disk was also easy to handle since each group had its own partitioned space for local storage.

We found that the stats were easier to gather at that level and we didn't have to parse the Operational Database to get the system usage during that timeframe.
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

You could NMON the box every two mins and dump that into splunk. Then report on the pid usage... but that would be overkill IMHO.


Yes QT, I suggest running a query on the ops console database directly. (datastage job to make that report of course)


That report would be easy enough to generate.

You could have any granularity you wanted.

duration of the job, quantity of executions, owning user id, project name, etc... Memory consumption...

You are most likely already doing disk charge back properly since that is an easier thing to track. They buy the mount, use it or not it's still the same charge back since it's reserved space. If you have shared scratch disk that gets nit picky to divide up and you might just want to waive that fee as a freebee for playing in the environment.
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

Thank you Andy and Paul for your replies. I will discuss these ideas with the team here!
Choose a job you love, and you will never have to work a day in your life. - Confucius
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

So what did you guys end up doing qt?

To the other audience members:

What are your chargeback models?


Here is my scenario:

I have a shared grid of compute nodes that needs to be put into a chargeback model. The Head Nodes are pretty much unique to each application team. The compute nodes are shared.

I can calculate the quantity of CPU consumed across all of the projects via DSODB (granted that does not contain any pre/post datastage shell scripting). I then take project A total CPU and obtain his environment percentage.

Just wondering what others are doing.

NMON PID output calculation?
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

I work in a government organization and they needed some guideline numbers quickly, in order to get allocations submitted for the next couple of fiscal year budget cycles.

I ended up doing "Plan A"--various calculations in Excel, as a one-time exercise, that may be repeated every 1-2 years as needed.

- The easy one was storage.

- Another was based on job counts per project (bin/dsjob -ljobs <project> | wc -l, for each project).

- Since we use ISD for web services, I had to factor those in too because they require extra configuration and they run 24x7 and require special monitoring.

- The last part was mostly manual, and a good use of Excel: I took all of our schedules into consideration, such as how frequently each sequence job runs as well as which ones run outside of normal working hours. This is the main factor affecting how much time our team is called to respond and fix things that break.

In the end we arrived at a fair way to split the costs across agencies. The total costs consist of 1) the annual IBM maintenance charges for software subscription and support, 2) the internal IT charges for hosting the LPARs, extra CPUs, extra memory, storage, and backups, and 3) the labor cost of having a team support all of it. So, Agency A will cover ##%, Agency B will cover ##%, and so on.

"Plan B" is that if any agency questions the calculations or demands something more granular for billing, then we can setup automated queries against the DSODB database or similar methods, and our time for setting that up would have to be charged out.
Choose a job you love, and you will never have to work a day in your life. - Confucius
Post Reply