Monitoring ETL Loads

jreddy · Post by **jreddy** » Tue Feb 27, 2018 1:09 pm

Wanted to see what options are out there in terms of Monitoring Loads. I have seen the following at various implementations - are there any others that are more efficient?

1) Job Scheduling tools - like Appworx that will send an alert if a job fails or is delayed
2) Monitoring Staff - watching the email notifications and report to the development team when loads fail or notifications are not received on time

Are there any other latest tools in the market that can interact with the ETL loads to send text message alerts or calls to the monitoring group? or any tool within the IBM suite that can do the same OR could we write some routines/processes in ETL to achieve any notifications other than email

Thanks !!

ray.wurlod · Post by **ray.wurlod** » Tue Feb 27, 2018 4:43 pm

If your jobs generate operational metadata, then you can see information such as row counts and execution date/time in data lineage reports.

Run history is stored in the database behind the Operations Console (this database is conventionally named DSODB).

Exceptions can be intercepted in Subscription Manager and cause an email to be sent or, more fancily, any other process (perhaps Stewardship Center, which is an implementation of IBM Business Process Manager over the top of IBM InfoSphere Information Server).

Just some ideas off the top of my head.

qt_ky · Post by **qt_ky** » Wed Feb 28, 2018 7:07 am

We did something very simple and made it part of our development standards, which was to always send sequence job abort notifications to a separate mailing list and always include "Aborted" in the subject line so that everyone receiving the emails could create mail rules to help alert them. Then we have some flexibility to choose who gets alerted--the original developers or some operations console or help desk staff, etc.

UCDI · Post by **UCDI** » Wed Feb 28, 2018 3:15 pm

we put an error handler in our jobs to mail a distribution list when the job fails. We also have a scheduler that emails. Just a simple addition to your top level sequence jobs ... error handler -> user variables (optional) -> email stage

jreddy · Post by **jreddy** » Wed Feb 28, 2018 6:03 pm

Yes, that was essentially what my option 2 in original email was - where email notifications are monitored, but this requires someone to keep monitoring those emails (for ABORTS or success notifications) instead of monitoring director. Is there a way to send other types of notifications like text alerts

qt_ky · Post by **qt_ky** » Thu Mar 01, 2018 8:05 am

If you want a more active type of monitoring, then yes, they could have staff monitor using Director which you have to install, but the the web-based Operations Console is also available. Just give them access and the URL.

As far as text alerts go, most wireless carriers have an email-to-text path available (at least the used to the last time I checked). It is often in the format of phone_number@something.com. You might want to check first how strict your security rules are. In our organization, the email header part was deemed to carry information that was too sensitive to send externally, such as server name or IP address. If your org doesn't care about security, then text away.

chulett · Post by **chulett** » Thu Mar 01, 2018 8:10 am

Exactly. At places I've worked where it was allowed (not here!) we had short "emails" that were sent as text messages to people's phones. And dare I even say it, pagers. Back in the day, of course.

jreddy · Post by **jreddy** » Thu Mar 01, 2018 3:40 pm

yes, being able to text would be the perfect solution, then it is more efficient as we can only look when we get the text message instead of having to stare at either Director or email. If we have work phones and security was not an issue - as it is just notifications on success/failure of loads, is there a quick and easier way to set up text alerting from DataStage?

ray.wurlod · Post by **ray.wurlod** » Thu Mar 01, 2018 8:50 pm

Subscription Center. Any user can set up subscriptions to be emailed when particular types of exception occur.

jreddy · Post by **jreddy** » Fri Mar 02, 2018 10:22 am

Thank you Ray, not sure I totally understand what a Subscription Center is - is that some sort of monitoring service we have to subscribe to (with vendors or vendor products) or is that something we have in our IBM suite of tools.. thanks

ray.wurlod · Post by **ray.wurlod** » Sun Mar 04, 2018 6:34 pm

Subscription Center is part of IBM InfoSphere Information Server.

ray.wurlod · Post by **ray.wurlod** » Sun Mar 04, 2018 6:36 pm

Most mail servers can be configured to convert email messages to SMS messages (provided the rules are followed), or even to pager messages.