Automated Code review/check

Do you have features you'd like to see in future releases of DataStage, MetaStage, Parameter Manager, Version Control or one of the other tools represented on this forum? Post your ideas here!

Moderators: chulett, rschirm

Post Reply
scorpion
Participant
Posts: 144
Joined: Thu May 12, 2005 4:19 am

Automated Code review/check

Post by scorpion »

Hi All,

I just wanted to know that:

Is there any utility or tool avilable to make the code review and cross check the naming standars or other checks in job ,which allows in automated way.

For ex: if I have 1000 jobs running in prod/dev and now as part of the COE i decided to check whether all those jobs are following the standards or best practises.

Instead of going and opening the individual job one by one ,any automated utility is avlable?

Thanks in advance.
Thanx&Regards
scorpion
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

No such tool is available. Some of us, myself included, have coded small applications to do such checking, but since each site has different standards that they wish enforced, such code is generally not re-usable.
PhilHibbs
Premium Member
Premium Member
Posts: 1044
Joined: Wed Sep 29, 2004 3:30 am
Location: Nottingham, UK
Contact:

Post by PhilHibbs »

I have an Excel spreadsheet with VBA that gathers a load of information about a DSX file. It's quite sophisticated and thorough, but a lot of it is hard-coded to the way we do things, and we only use it for Server jobs. It isn't the best, most structured piece of code in the world ether. I'll share a copy with anyone who is interested, but I don't have anywhere to upload it so it will have to be via private email.

It gathers:
  • Job category, name, type, timestamp, change history (in our own convention)
    SAP connection details
    SAP ABAP extract program names
    SAP IDocs
    Parameters & default values
    File stage details and file names (expanded with default parameter values)
    File schemas
    Version information (again, using our own convention), and a function to compare version info with an analysis of another project
    CRC of all constraints, derivations, and lookup keys
All information is stored in spreadsheet tabs. It takes progressively longer the more jobs you have, at around 2000 it can get very slow, so I tend to cut my DSX files first.

I need to add Stage Variable gathering as well, so that those can be checked against naming standards.

Search for "DataStage Analysis spreadsheet" to find the thread dedicated to this tool, my last post in that thread will have a download link for the latest version.
Last edited by PhilHibbs on Mon Aug 02, 2010 3:27 am, edited 1 time in total.
Phil Hibbs | Capgemini
Technical Consultant
kduke
Charter Member
Charter Member
Posts: 5227
Joined: Thu May 29, 2003 9:47 am
Location: Dallas, TX
Contact:

Post by kduke »

Good idea. I would like to see the code.
Mamu Kim
PhilHibbs
Premium Member
Premium Member
Posts: 1044
Joined: Wed Sep 29, 2004 3:30 am
Location: Nottingham, UK
Contact:

Post by PhilHibbs »

PhilHibbs wrote:I need to add Stage Variable gathering as well, so that those can be checked against naming standards.
This is now done, stage variables are gathered onto a tab with a count of the number of references to each, as with job parameters. PM me with your email address if you are interested in this project analysis tool.
Phil Hibbs | Capgemini
Technical Consultant
kduke
Charter Member
Charter Member
Posts: 5227
Joined: Thu May 29, 2003 9:47 am
Location: Dallas, TX
Contact:

Post by kduke »

Got it. Thanks.
Mamu Kim
JoshGeorge
Participant
Posts: 612
Joined: Thu May 03, 2007 4:59 am
Location: Melbourne

Post by JoshGeorge »

XMETA (repository) provides this information in different tables. You can check there only by writing queries or get the information into a spreadsheet to generate a report with the standards defined there.
In the table name like DATASTAGEX%*%DSJOBDEFC2E76D84 you will get

CATEGORY_XMETA (For folder name)
NAME_XMETA (For job name))
DSJOBTYPE_XMETA (For job type) [ 0 is Server job, 2 is Sequencer, 3 parallel job ...]
In the table name like DATASTAGEX%*%DSSTAGEC2E76D84 you will get

NAME_XMETA (For stage names)
Joshy George
<a href="http://www.linkedin.com/in/joshygeorge1" ><img src="http://www.linkedin.com/img/webpromo/bt ... _80x15.gif" width="80" height="15" border="0"></a>
abc123
Premium Member
Premium Member
Posts: 605
Joined: Fri Aug 25, 2006 8:24 am

Post by abc123 »

Josh, I don't see a table called

DATASTAGEXNAME_XMETADSJOBDEFC2E76D84

for all job names?
rohitash_saini
Participant
Posts: 4
Joined: Wed May 08, 2019 2:21 am

DataStage Analysis spreadsheet

Post by rohitash_saini »

Phil,
Can you please share this spreadsheet ?
Rohitash
PhilHibbs
Premium Member
Premium Member
Posts: 1044
Joined: Wed Sep 29, 2004 3:30 am
Location: Nottingham, UK
Contact:

Re: DataStage Analysis spreadsheet

Post by PhilHibbs »

rohitash_saini wrote:Can you please share this spreadsheet ?
Done, link updated in the first post of the main thread. Dropbox changed the way public shared files worked, it's now on Google Drive. If anyone has a better place to host it, please feel free to "mirror" it and I will include a link in my post.
Phil Hibbs | Capgemini
Technical Consultant
stuartjvnorton
Participant
Posts: 527
Joined: Thu Apr 19, 2007 1:25 am
Location: Melbourne

Post by stuartjvnorton »

MettleCI offers what it calls Compliance checking as part of its CICD processes, and it can also be run standalone through the Workbench.
It has a number of pre-canned rules to check things like naming standards, use of deprecated stages, adherence to best practices (no hardcoded DB credentials or file names, link sorts), etc. You can also write your own using Gremlin.

https://www.mettleci.com/mettleci.html#videos
rohitash_saini
Participant
Posts: 4
Joined: Wed May 08, 2019 2:21 am

Re: DataStage Analysis spreadsheet

Post by rohitash_saini »

Thank you Phil.
Rohitash
Post Reply