git/RTC/ISM with Datastage projects

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
dsguyok
Premium Member
Premium Member
Posts: 24
Joined: Thu Jan 21, 2010 10:22 pm

git/RTC/ISM with Datastage projects

Post by dsguyok »

Hi

We are looking to add a "continuous delivery" model into our current SDLC. We are about to choose a new version control system. We are currently looking at git and RTC.

Does anyone have any good or bad experience with Datastage projects and git? git seems to be an ok SCM tool but just wondering if the IBM product RTC has any special integration features for Datastage that git doesn't.

Also I would like to automate our deployment process as much as possible (it's currently very manual). I understand that Information Server Manager (ISM) is the IBM-backed deployment tool for Datastage projects. Are there any suggestions for alternatives to ISM?

I have worked with RTC and ISM a little bit on previous projects. Just wondering if ISM would play as nicely with git as it does with RTC?

This IBM page suggests that CVS, RTC and Clearcase are the only options for linking to ISM. But I saw someone else say on here that they have linked TFS to ISM.

Thanks.
Mike
Premium Member
Premium Member
Posts: 1021
Joined: Sun Mar 03, 2002 6:01 pm
Location: Tampa, FL

Post by Mike »

CVS, RTC, and Clearcase are likely supported out-of-the-box with ISM (RTC must be newly supported in 9.1). You can use any tool that provides an eclipse-based plugin. I personally prefer Subversion (an open source solution using the Subclipse plugin), but they will all operate very similarly via their eclipse-based plugins.

I don't have any knowledge of git... you'll have to see if there is an eclipse-based plugin for it.

As an alternative to ISM, you can always roll your own solution leveraging some of the various command line tools (e.g. istool), but in my mind that is a step backward. If you don't already have a pre-existing solution from the pre-ISM days, I would recommend sticking with ISM. It's likely that IBM will continue enhancing ISM in future releases.

Mike
jhmckeever
Premium Member
Premium Member
Posts: 301
Joined: Thu Jul 14, 2005 10:27 am
Location: Melbourne, Australia
Contact:

Post by jhmckeever »

It's an old post but I'll comment anyway in case the issue is relevant to anyone else with the same question:

We've created a custom menu item in the DataStage Designer client which invokes a checkin-process.

The process is execute by Atlassian Bamboo (a build automation tool similar to Jenkins or Hudson) which extracts the currently open DataStage job from XMETA, invokes a set of unit tests, then checks the job into Stash (Atlassian's wrapped version of Git). The unit tests involve loading the DataStage job's OSH into a graph database and testing for anti-patterns, such as adjacent transformers, or anything else we want to identify. Some lead to warnings, some lead to the check-in being rejected. Bamboo can additionally update the job's long description with version/deployment information.

Jobs are then automatically deployed to an integration test environment (again using Bamboo) where they are tested against restricted data volumes with functional test cases. When all tests are passed the job becomes a candidate for promotion to production, or other test environments as necessary.

This process means all developers work in the same project, and we have good control over, and transparency of, our code releases.
<b>John McKeever</b>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
jhmckeever
Premium Member
Premium Member
Posts: 301
Joined: Thu Jul 14, 2005 10:27 am
Location: Melbourne, Australia
Contact:

Post by jhmckeever »

Those following this thread might be interested to know that my organisation's solution for implementing DevOps approaches with DataStage development (called 'MettleCI') has now been announced on the Atlassian Marketplace:

https://marketplace.atlassian.com/plugi ... .mettleci/

You can find out more here: https://www.mettleci.com

Image
<b>John McKeever</b>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
Post Reply