DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
Madhumitha_Raghunathan



Group memberships:
Premium Members

Joined: 22 Apr 2011
Posts: 59

Points: 517

Post Posted: Tue May 08, 2018 12:27 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Additional info: AIX 7.1, Power HA Version 7
Hi,
We have IIS 11.5 is installed on active / passive cluster (Power HA Version 7). When the application fails over to secondary host, and engine starts up more than 45k DB connections are opened to XMETA (sample below) and server crashes due to memory spike to 100% and swap to 99%. The application comes up fine on the primary host and we dont have that many internal connections and it is stable. All the tiers are installed on one single host.

tcp4 0 0 iis-uat.db2c_db2 iis-uat.33010 ESTABLISHED
tcp 0 0 iis-uat.33010 iis-uat.db2c_db2 ESTABLISHED
tcp4 0 0 iis-uat.db2c_db2 iis-uat.33013 ESTABLISHED
tcp 0 0 iis-uat.33013 iis-uat.db2c_db2 ESTABLISHED
tcp4 0 0 iis-uat.db2c_db2 iis-uat.33016 ESTABLISHED
tcp 0 0 iis-uat.33016 iis-uat.db2c_db2 ESTABLISHED

Would be great if someone could help. Appreciate it.

_________________
Thanks,
Madhumitha
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1254

Points: 8241

Post Posted: Tue May 08, 2018 2:37 pm Reply with quote    Back to top    

Is this your first failover?
Rate this response:  
Not yet rated
Madhumitha_Raghunathan



Group memberships:
Premium Members

Joined: 22 Apr 2011
Posts: 59

Points: 517

Post Posted: Tue May 08, 2018 2:46 pm Reply with quote    Back to top    

Yes. another thing that I forgot to mention is that the repository was installed first IBM DB2 and the databases and objects were created with the scripts provided by IBM. The Engine and services were installed from the IIS installation pkg after HACMP was setup.

_________________
Thanks,
Madhumitha
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1254

Points: 8241

Post Posted: Wed May 09, 2018 7:58 am Reply with quote    Back to top    

When the DB fails over, you will still have to restart WAS and DataStage. Did you do that?

The nature of the failover... Is the database still accessible by the name defined in Version.xml and the DB2 Catalog?
Rate this response:  
Not yet rated
Madhumitha_Raghunathan



Group memberships:
Premium Members

Joined: 22 Apr 2011
Posts: 59

Points: 517

Post Posted: Wed May 09, 2018 8:06 am Reply with quote    Back to top    

Yes. Our start order is always DB2, WAS, NodeAgent and Engine. Also once all of these tiers startup, I am able to login to datastage also. But the number of connections increase drastically after 5 minutes and the failover server crashes due to memory and paging space issue. We have 48 GB of memory on each box. The whole process takes up only 5 GB of memory on the primary server but even 48 GB is not enough on failover.

_________________
Thanks,
Madhumitha
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1254

Points: 8241

Post Posted: Wed May 09, 2018 4:00 pm Reply with quote    Back to top    

Ahh so the failover was as a result of a broken process and not a planned failover test?
Rate this response:  
Not yet rated
Madhumitha_Raghunathan



Group memberships:
Premium Members

Joined: 22 Apr 2011
Posts: 59

Points: 517

Post Posted: Thu May 10, 2018 6:55 am Reply with quote    Back to top    

When IIS is up and running on the primary host it is all good and doesn't consume much memory (about 5 GB out of available 48GB). We are trying to switch to the secondary server to complete a maintenance activity on the primary host and not because of a broken process. When we failover, we are moving the resource groups to the failover box and the application starts up fine and I am able to login as well. After about 5 minutes the failover server crashes as it exceed the 48GB due to the looping DB connections.

_________________
Thanks,
Madhumitha
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1254

Points: 8241

Post Posted: Thu May 10, 2018 7:51 am Reply with quote    Back to top    

Get IBM engaged.

We won't be able to do much from here since we are not poking on your system.

I would still try to look at Version.xml to see if you can communicated properly to your xmeta and the entry that is cataloged in db2.

You should run your check xmeta script to see if that communication path is good.

$DSHOME/../../ASBServer/bin/AppServerAdmin.sh -w
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours