DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
rsampath



Group memberships:
Premium Members

Joined: 25 Sep 2018
Posts: 2

Points: 57

Post Posted: Tue Oct 30, 2018 7:46 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Additional info: IGC 11.5
Hi all,

Apologies, in advance, for the long post.

Looking for some guidance on setting up our IGC environment (11.5). We're planning to use IGC to get metadata in from various systems in our landscape:

- DataStage/QualityStage/Info. Analyzer - Automatic
- DB (SQL Server/DB2/SAPHANA) - IMAM (ODBC/JDBC or CSV connectors (export))
- IBM Replication (IIDR) - Import via IIDR-IGC project ( Github )
- Hadoop - IMAM (HDFS/Hive or IGC-Atlas connector (we're on HDP 2.6.2; Atlas 0.8.0))
- Cognos BI - IMAM

I've reviewed other posts in this forum on setting up the IGC environment (should separate from core ETL environment (DEV/QA/Staging/PROD)):
- Post 1
- Post 2

I've also reviewed IBM Redbook SG247939 that discusses this topic, in some detail.

My current thinking is to have 2 environments (1 Sandbox & 1 production), each with its own IGC (& IMAM & Metadata interchange servers) environment, for the following reasons:
- To test out connectors/drivers before installing them for the production IGC instance
- To build/test imports/scripting/metadata import automation before implementing them in production

The publishing (surfacing out) of technical metadata as well as mapping of technical to business metadata would occur in the production instance.

Is this a typical environment setup? Is there normally 1 metadata interchange server per environment, or are there multiple? What are the factors that go into this decision?

If unable to have a separate IGC environment, has anyone setup the IGC environment to be part of their mainstream ETL environment? The main benefit of doing this, that I see, is to eliminate the need to copy the DS/QS jobs from the mainstream to the IGC environment, to load that metadata. What are the risks/issues associated with this approach?

Thanks, in advance.

RS
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2868
Location: USA
Points: 21784

Post Posted: Mon Jan 14, 2019 10:30 am Reply with quote    Back to top    

You can also use your sandbox environment to pre-test all new feature packs/fix packs, service packs, upgrades, and security patches. Sounds like a good plan.

We have Info Server Enterprise Edition with a mature DataStage and QualityStage workload, with all server tiers on one server. This supports IGC as well. We have an environment for production and another for non-production and another that we call sandbox, which is reserved for admins for the purposed listed above.

Way, way back, many major releases ago on version 8.5, we encountered some security risks in that some business glossary were able to view detailed DataStage job logs and glean information like computer names, user names, etc. But the security model evolved rapidly and I have not seen any risks since version 8.7 and higher. We are not seeing any noticeable impact from IGC activity on the ETL workload either.

Initially as our IGC usage grows, we plan to stay as is. At some point, as you must have seen in IBM documentation, they recommend standing up your IGC instances for authoring and for having a published catalog as separate installations. I don't think we have the need to do so yet. Hope that helps!

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2868
Location: USA
Points: 21784

Post Posted: Mon Jan 14, 2019 10:39 am Reply with quote    Back to top    

Moderator, the original posts' links 1 and 2 are missing the www parts in the URLs, which cause topics to be blocked because dsxchange thinks you're not logged in... Wondering if you wanted to patch up those URLs? Thank you.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42997
Location: Denver, CO
Points: 221838

Post Posted: Mon Jan 14, 2019 10:51 am Reply with quote    Back to top    

You betcha.

_________________
-craig

I swore that I would love you to the end of time, so now I'm praying for the end of time to hurry up and arrive
'Cause if I gotta spend another minute with you I don't think that I can really survive
Rate this response:  
Not yet rated
rsampath



Group memberships:
Premium Members

Joined: 25 Sep 2018
Posts: 2

Points: 57

Post Posted: Mon Jan 14, 2019 11:44 am Reply with quote    Back to top    

Thank you for your response.

To keep our landscape simple, in the initial term, we've decided to maintain 2 instances - non-PROD & PROD.

As we get more mature in this space, and when the need for an upgrade arises, the sandbox makes a lot of sense, to avoid impacting the mainstream environments. I'll definitely keep that in mind.

Thanks, once again.
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2868
Location: USA
Points: 21784

Post Posted: Tue Jan 15, 2019 3:47 am Reply with quote    Back to top    

Sure thing.

Don't know if your environment supports this or not... With our hardware and virtualization, we have been able to build multiple LPARs within the same shared CPU pools, and avoid licensing additional PVUs.

For example, our dev/test server's LPAR resides in the same CPU pool as our sandbox server's LPAR. The CPU pool is covered by our non-production Info Server PVU licenses. The drawback of course, is that the workload of one LPAR impacts the workload of other LPARs that share the same CPUs. But it works well for us.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3824

Points: 30832

Post Posted: Tue Jan 15, 2019 9:04 am Reply with quote    Back to top    

Great discussion. It is worth noting also that a lot of sites will have a separate instance of Information Server for IGC and governance activities. This is primarily because the upgrade effort to ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours