Source control for datastage

nzeidat · Post by **nzeidat** » Fri Dec 09, 2016 9:25 am

Hi,
I wonder if any member on this group has experience or lessons learned with respect to Source Control for datastage. We are tasked to explore options and tools out there. We are currently evaluating using Subversion (SVN) for Source Control and Configuration management. We are exploring 2 methods
1) Using SVN with export files (.dsx) using the Trunk/Tag/Branch model
2) Using SVN through IBM Information Server manager.
If you have explored these options or other options for Configuration Management & Source Control, please share your experiences.

Thanks a lot,
- nzeidat

UCDI · Post by **UCDI** » Fri Dec 09, 2016 12:18 pm

i will warn you that export files need a hand-rolled comparison program. If, for example, you drop an export from your production server and an export from your dev server together and do a comparison, they WILL NOT match even if the "code" is the same. A number of fields will be different. I don't know your product and if it can be configured to handle this easily or not.

PaulVL · Post by **PaulVL** » Mon Dec 12, 2016 12:53 pm

I also recommend doing it from the server side using isx files for your subversion checkin (with executable).

That way you can automate your code deployment via your repository using the istool command.

jhmckeever · Post by **jhmckeever** » Thu Jan 12, 2017 5:55 pm

We've created a custom menu item in the DataStage Designer client which invokes a checkin-process.

The process is execute by Atlassian Bamboo (a build automation tool similar to Jenkins or Hudson) which extracts the currently open DataStage job from XMETA, invokes a set of unit tests, then checks the job into Stash (Atlassian's wrapped version of Git). The unit tests involve loading the DataStage job's OSH into a graph database and testing for anti-patterns, such as adjacent transformers, or anything else we want to identify. Some lead to warnings, some lead to the check-in being rejected. Bamboo can additionally update the job's long description with version/deployment information.

Jobs are then automatically deployed to an integration test environment (again using Bamboo) where they are tested against restricted data volumes with functional test cases. When all tests are passed the job becomes a candidate for promotion to production, or other test environments as necessary.

This process means all developers work in the same project, and we have good control over, and transparency of, our code releases.