DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
truenorth
Participant



Joined: 18 Jan 2010
Posts: 128
Location: San Antonio
Points: 861

Post Posted: Tue Sep 11, 2018 11:16 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Additional info: Best practices, recommendations
I'm looking for recommendations and best practices on backing up IADB.

This is our situation:
- We have been using IA for three years.
- IADB has grown to 10TB.
- We backed it up once and it took 50 hours.
- Most, if not all, projects do not have auto purge set in Project Properties.
- IBM provided an external PURGE script, which we plan to execute soon. We will retain 13 months worth of data.
- Our IT team says that IA needs to be shut down to conduct any backup. This conflicts with the fact that we run IA jobs 24/7 including on weekends.

How have you approached IADB backup in your organization? Based on the above, what do you recommend?

_________________
Todd Ramirez
Sr Consultant, Data Quality
San Antonio TX
asorrell
Site Admin

Group memberships:
Premium Members, DSXchange Team, Inner Circle, Server to Parallel Transition Group

Joined: 04 Apr 2003
Posts: 1683
Location: Colleyville, Texas
Points: 22862

Post Posted: Tue Sep 11, 2018 8:17 pm Reply with quote    Back to top    

I would recommend doing the purge in very small chunks initially to determine timings / impact. The admin at a previous site initially kicked it off to purge an entire month and the purge process not only killed the system performance, it took forever to complete.

We changed it to purge a day at a time which only took a few hours for each run. We used a script so it was semi-automated and we could stop it as required to insure the critical SLA times weren't impacted.

The DBA's want the database to be inactive so the backup is of a database in a good state, with all transactions committed. It's hard to argue with that because it means the restore would be to a "good" state.

If your shop is running 24/7 with no scheduled maintenance windows then I'm surprised you haven't seen issues with corruption impacting your applications already.

Your organization needs to discuss making the necessary downtime mandatory - both for maintenance (DataStage and database) and backups. It may be painful to do so, but the impact of a corrupted, non-restorable database could be catastrophic.

_________________
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2017
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2818
Location: USA
Points: 21341

Post Posted: Wed Sep 12, 2018 4:22 am Reply with quote    Back to top    

We do automated weekly offline backups, meaning that we have built scripts to stop all the server processes, perform database backups, and start everything up again. I would like to switch to online backups if possible.

We also create a separate IADB per IA project, mainly so that as or when each project ends, its IADB can be cleanly dropped without impacting other IA projects and without the headache of figuring out how to clean up storage for one IA project that would have been mixed in with a bunch of others. If you went that route also, then it may help you run multiple backups simultaneously or at least enable some flexibility with running backups on different schedules.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
truenorth
Participant



Joined: 18 Jan 2010
Posts: 128
Location: San Antonio
Points: 861

Post Posted: Thu Sep 13, 2018 9:24 am Reply with quote    Back to top    

A million thanks, everyone. I will bring all your brilliant suggestions to our team.

Andy,
The person in charge of the purge plan to start with anything older than 3 years, then 2.5, etc. You said one month took forever and you had to go to one day at a time and it still took a few hours. I will communicate your experience to him.

qt_ky,
We're moving from 11.5 to 11.7 and I will suggest a separate IADB for each project.

Re online backups, our IT team says it can't be done because of constant "DML activity". I don't know myself, but if you pull it off, I'd be interested to know.

Thanks again!

_________________
Todd Ramirez
Sr Consultant, Data Quality
San Antonio TX
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42762
Location: Denver, CO
Points: 220350

Post Posted: Thu Sep 13, 2018 11:56 am Reply with quote    Back to top    

What's your database? Curious for the whole open/hot versus closed/cold backup part of this discussion.

_________________
-craig

Research shows that 6 out of 7 dwarves aren't happy
Rate this response:  
Not yet rated
truenorth
Participant



Joined: 18 Jan 2010
Posts: 128
Location: San Antonio
Points: 861

Post Posted: Tue Sep 18, 2018 7:00 am Reply with quote    Back to top    

It's DB2, chulett.

_________________
Todd Ramirez
Sr Consultant, Data Quality
San Antonio TX
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42762
Location: Denver, CO
Points: 220350

Post Posted: Tue Sep 18, 2018 7:43 am Reply with quote    Back to top    

Ah, well then never mind. Pondered chiming in if it was Oracle but have no experience with DB2.

_________________
-craig

Research shows that 6 out of 7 dwarves aren't happy
Rate this response:  
Not yet rated
truenorth
Participant



Joined: 18 Jan 2010
Posts: 128
Location: San Antonio
Points: 861

Post Posted: Wed Sep 19, 2018 12:12 pm Reply with quote    Back to top    

No worries, Craig.

_________________
Todd Ramirez
Sr Consultant, Data Quality
San Antonio TX
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours