Backing up IADB

This forum contains ProfileStage posts and now focuses at newer versions Infosphere Information Analyzer.

Moderators: chulett, rschirm

Post Reply
truenorth
Participant
Posts: 139
Joined: Mon Jan 18, 2010 4:59 pm
Location: San Antonio

Backing up IADB

Post by truenorth »

I'm looking for recommendations and best practices on backing up IADB.

This is our situation:
- We have been using IA for three years.
- IADB has grown to 10TB.
- We backed it up once and it took 50 hours.
- Most, if not all, projects do not have auto purge set in Project Properties.
- IBM provided an external PURGE script, which we plan to execute soon. We will retain 13 months worth of data.
- Our IT team says that IA needs to be shut down to conduct any backup. This conflicts with the fact that we run IA jobs 24/7 including on weekends.

How have you approached IADB backup in your organization? Based on the above, what do you recommend?
Todd Ramirez
Sr Consultant, Data Quality
San Antonio TX
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

I would recommend doing the purge in very small chunks initially to determine timings / impact. The admin at a previous site initially kicked it off to purge an entire month and the purge process not only killed the system performance, it took forever to complete.

We changed it to purge a day at a time which only took a few hours for each run. We used a script so it was semi-automated and we could stop it as required to insure the critical SLA times weren't impacted.

The DBA's want the database to be inactive so the backup is of a database in a good state, with all transactions committed. It's hard to argue with that because it means the restore would be to a "good" state.

If your shop is running 24/7 with no scheduled maintenance windows then I'm surprised you haven't seen issues with corruption impacting your applications already.

Your organization needs to discuss making the necessary downtime mandatory - both for maintenance (DataStage and database) and backups. It may be painful to do so, but the impact of a corrupted, non-restorable database could be catastrophic.
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

We do automated weekly offline backups, meaning that we have built scripts to stop all the server processes, perform database backups, and start everything up again. I would like to switch to online backups if possible.

We also create a separate IADB per IA project, mainly so that as or when each project ends, its IADB can be cleanly dropped without impacting other IA projects and without the headache of figuring out how to clean up storage for one IA project that would have been mixed in with a bunch of others. If you went that route also, then it may help you run multiple backups simultaneously or at least enable some flexibility with running backups on different schedules.
Choose a job you love, and you will never have to work a day in your life. - Confucius
truenorth
Participant
Posts: 139
Joined: Mon Jan 18, 2010 4:59 pm
Location: San Antonio

Post by truenorth »

A million thanks, everyone. I will bring all your brilliant suggestions to our team.

Andy,
The person in charge of the purge plan to start with anything older than 3 years, then 2.5, etc. You said one month took forever and you had to go to one day at a time and it still took a few hours. I will communicate your experience to him.

qt_ky,
We're moving from 11.5 to 11.7 and I will suggest a separate IADB for each project.

Re online backups, our IT team says it can't be done because of constant "DML activity". I don't know myself, but if you pull it off, I'd be interested to know.

Thanks again!
Todd Ramirez
Sr Consultant, Data Quality
San Antonio TX
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

What's your database? Curious for the whole open/hot versus closed/cold backup part of this discussion.
-craig

"You can never have too many knives" -- Logan Nine Fingers
truenorth
Participant
Posts: 139
Joined: Mon Jan 18, 2010 4:59 pm
Location: San Antonio

Post by truenorth »

It's DB2, chulett.
Todd Ramirez
Sr Consultant, Data Quality
San Antonio TX
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Ah, well then never mind. Pondered chiming in if it was Oracle but have no experience with DB2.
-craig

"You can never have too many knives" -- Logan Nine Fingers
truenorth
Participant
Posts: 139
Joined: Mon Jan 18, 2010 4:59 pm
Location: San Antonio

Post by truenorth »

No worries, Craig.
Todd Ramirez
Sr Consultant, Data Quality
San Antonio TX
truenorth
Participant
Posts: 139
Joined: Mon Jan 18, 2010 4:59 pm
Location: San Antonio

Post by truenorth »

Follow up questions:

What is your SLA on IADB backup?

What is your Return to Service (RTS), i.e., in case of a disaster, how soon should IA be back in operation?

What is your backup frequency? In other words, how old would restored data be upon recovery?
Todd Ramirez
Sr Consultant, Data Quality
San Antonio TX
Post Reply