Monitoring ETL Loads
Moderators: chulett, rschirm, roy
Monitoring ETL Loads
Wanted to see what options are out there in terms of Monitoring Loads. I have seen the following at various implementations - are there any others that are more efficient?
1) Job Scheduling tools - like Appworx that will send an alert if a job fails or is delayed
2) Monitoring Staff - watching the email notifications and report to the development team when loads fail or notifications are not received on time
Are there any other latest tools in the market that can interact with the ETL loads to send text message alerts or calls to the monitoring group? or any tool within the IBM suite that can do the same OR could we write some routines/processes in ETL to achieve any notifications other than email
Thanks !!
1) Job Scheduling tools - like Appworx that will send an alert if a job fails or is delayed
2) Monitoring Staff - watching the email notifications and report to the development team when loads fail or notifications are not received on time
Are there any other latest tools in the market that can interact with the ETL loads to send text message alerts or calls to the monitoring group? or any tool within the IBM suite that can do the same OR could we write some routines/processes in ETL to achieve any notifications other than email
Thanks !!
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
If your jobs generate operational metadata, then you can see information such as row counts and execution date/time in data lineage reports.
Run history is stored in the database behind the Operations Console (this database is conventionally named DSODB).
Exceptions can be intercepted in Subscription Manager and cause an email to be sent or, more fancily, any other process (perhaps Stewardship Center, which is an implementation of IBM Business Process Manager over the top of IBM InfoSphere Information Server).
Just some ideas off the top of my head.
Run history is stored in the database behind the Operations Console (this database is conventionally named DSODB).
Exceptions can be intercepted in Subscription Manager and cause an email to be sent or, more fancily, any other process (perhaps Stewardship Center, which is an implementation of IBM Business Process Manager over the top of IBM InfoSphere Information Server).
Just some ideas off the top of my head.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
We did something very simple and made it part of our development standards, which was to always send sequence job abort notifications to a separate mailing list and always include "Aborted" in the subject line so that everyone receiving the emails could create mail rules to help alert them. Then we have some flexibility to choose who gets alerted--the original developers or some operations console or help desk staff, etc.
Choose a job you love, and you will never have to work a day in your life. - Confucius
Yes, that was essentially what my option 2 in original email was - where email notifications are monitored, but this requires someone to keep monitoring those emails (for ABORTS or success notifications) instead of monitoring director. Is there a way to send other types of notifications like text alerts
If you want a more active type of monitoring, then yes, they could have staff monitor using Director which you have to install, but the the web-based Operations Console is also available. Just give them access and the URL.
As far as text alerts go, most wireless carriers have an email-to-text path available (at least the used to the last time I checked). It is often in the format of phone_number@something.com. You might want to check first how strict your security rules are. In our organization, the email header part was deemed to carry information that was too sensitive to send externally, such as server name or IP address. If your org doesn't care about security, then text away.
As far as text alerts go, most wireless carriers have an email-to-text path available (at least the used to the last time I checked). It is often in the format of phone_number@something.com. You might want to check first how strict your security rules are. In our organization, the email header part was deemed to carry information that was too sensitive to send externally, such as server name or IP address. If your org doesn't care about security, then text away.
Choose a job you love, and you will never have to work a day in your life. - Confucius
yes, being able to text would be the perfect solution, then it is more efficient as we can only look when we get the text message instead of having to stare at either Director or email. If we have work phones and security was not an issue - as it is just notifications on success/failure of loads, is there a quick and easier way to set up text alerting from DataStage?
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: