Anyone using Splunk to mine info from DataStage?

PaulVL · Post by **PaulVL** » Thu Oct 09, 2014 12:57 pm

Hi, I was wondering if anyone wrote the parser needed for the Operational Metadata generated XML files to feed splunk.

Also, since IBM bought Platform LSF and it is a recommended Grid Resource Manager for DataStage (should you chose to go grid), has anyone written a splunk parser for the LSF logs? The splunk site has a mention of this, but it's basically another user asking if anyone has done it yet.

Thanks in advance.

PaulVL · Post by **PaulVL** » Mon Jan 26, 2015 12:11 pm

Bump.

Still looking to see if anyone out there has paired Splunk + DataStage.

Thanks

skathaitrooney · Post by **skathaitrooney** » Tue Nov 29, 2016 3:54 am

Paul, were you able to integrate Splunk with DataStage ?

anamika · Post by **anamika** » Tue Nov 29, 2016 7:57 am

Paul, skatha,
At our Datastage site, we are/have extracting required information from DS logs - however, we are not using splunk/ELK or other log parsers. We are using perl/awk and similar tools to do this.
Is there something specific that you are looking for ?

Thx

eostic · Post by **eostic** » Tue Nov 29, 2016 12:16 pm

How about just using DS with xmlInput Stage to parse them? They are very simple xmls...

Ernie

PaulVL · Post by **PaulVL** » Tue Nov 29, 2016 4:44 pm

it's not the datastage side that was of any issue. It is the coupling of both products that I needed assistance on.

And no, I have not had the time to get splunk feeding off of the LSF logs.
I do not want to make a datastage log parser since the act of extracting a log is something I want to avoid.
Creating a datastage job to parse the XML files when you have 20K jobs running per day is also something I do not want to do since that means a ton more datastage jobs, and as a result could create an invinite loop if not coded properly.

So basically nobody out there is using splunk to mine any datastage metadata it would seem.

I have splunk set up for the NMON data dashboards. But something more akin to DataStage jobs would have been nice.

DSODB can be a source of the data, but the dashboards it presents are not 100% what I am looking for.

Would be nice to get a GANNT chart view of a daily project run. Sequencers + jobs with visual dependencies.

I might have to try and code something with highcharts and a table dump from DSODB. That stuff is just not my strong suite, I've managed to get by on my good looks alone so far.

chulett · Post by **chulett** » Tue Nov 29, 2016 5:23 pm