Page 1 of 1

Monitoring ETL Loads

Posted: Tue Feb 27, 2018 1:09 pm
by jreddy
Wanted to see what options are out there in terms of Monitoring Loads. I have seen the following at various implementations - are there any others that are more efficient?

1) Job Scheduling tools - like Appworx that will send an alert if a job fails or is delayed
2) Monitoring Staff - watching the email notifications and report to the development team when loads fail or notifications are not received on time

Are there any other latest tools in the market that can interact with the ETL loads to send text message alerts or calls to the monitoring group? or any tool within the IBM suite that can do the same OR could we write some routines/processes in ETL to achieve any notifications other than email

Thanks !!

Posted: Tue Feb 27, 2018 4:43 pm
by ray.wurlod
If your jobs generate operational metadata, then you can see information such as row counts and execution date/time in data lineage reports.

Run history is stored in the database behind the Operations Console (this database is conventionally named DSODB).

Exceptions can be intercepted in Subscription Manager and cause an email to be sent or, more fancily, any other process (perhaps Stewardship Center, which is an implementation of IBM Business Process Manager over the top of IBM InfoSphere Information Server).

Just some ideas off the top of my head.

Posted: Wed Feb 28, 2018 7:07 am
by qt_ky
We did something very simple and made it part of our development standards, which was to always send sequence job abort notifications to a separate mailing list and always include "Aborted" in the subject line so that everyone receiving the emails could create mail rules to help alert them. Then we have some flexibility to choose who gets alerted--the original developers or some operations console or help desk staff, etc.

Posted: Wed Feb 28, 2018 3:15 pm
by UCDI
we put an error handler in our jobs to mail a distribution list when the job fails. We also have a scheduler that emails. Just a simple addition to your top level sequence jobs ... error handler -> user variables (optional) -> email stage

Posted: Wed Feb 28, 2018 6:03 pm
by jreddy
Yes, that was essentially what my option 2 in original email was - where email notifications are monitored, but this requires someone to keep monitoring those emails (for ABORTS or success notifications) instead of monitoring director. Is there a way to send other types of notifications like text alerts

Posted: Thu Mar 01, 2018 8:05 am
by qt_ky
If you want a more active type of monitoring, then yes, they could have staff monitor using Director which you have to install, but the the web-based Operations Console is also available. Just give them access and the URL.

As far as text alerts go, most wireless carriers have an email-to-text path available (at least the used to the last time I checked). It is often in the format of phone_number@something.com. You might want to check first how strict your security rules are. In our organization, the email header part was deemed to carry information that was too sensitive to send externally, such as server name or IP address. If your org doesn't care about security, then text away.

Posted: Thu Mar 01, 2018 8:10 am
by chulett
Exactly. At places I've worked where it was allowed (not here!) we had short "emails" that were sent as text messages to people's phones. And dare I even say it, pagers. Back in the day, of course. :wink:

Posted: Thu Mar 01, 2018 3:40 pm
by jreddy
yes, being able to text would be the perfect solution, then it is more efficient as we can only look when we get the text message instead of having to stare at either Director or email. If we have work phones and security was not an issue - as it is just notifications on success/failure of loads, is there a quick and easier way to set up text alerting from DataStage?

Posted: Thu Mar 01, 2018 8:50 pm
by ray.wurlod
Subscription Center. Any user can set up subscriptions to be emailed when particular types of exception occur.

Posted: Fri Mar 02, 2018 10:22 am
by jreddy
Thank you Ray, not sure I totally understand what a Subscription Center is - is that some sort of monitoring service we have to subscribe to (with vendors or vendor products) or is that something we have in our IBM suite of tools.. thanks

Posted: Sun Mar 04, 2018 6:34 pm
by ray.wurlod
Subscription Center is part of IBM InfoSphere Information Server.

Posted: Sun Mar 04, 2018 6:36 pm
by ray.wurlod
Most mail servers can be configured to convert email messages to SMS messages (provided the rules are followed), or even to pager messages.