Here's some Perl code for Parameter replacement

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
jdmiceli
Premium Member
Premium Member
Posts: 309
Joined: Wed Feb 22, 2006 10:03 am
Location: Urbandale, IA

Here's some Perl code for Parameter replacement

Post by jdmiceli »

Hi all!

I'm not sure if this is the proper forum for this message, but I figured what I'm putting up here could be used across the different DataStage platforms.

We have recently acquired DataStage for our company and we already have seven different teams that have jumped on the development bandwagon. As most of you know, several of the major frustrations of this process are beating the learning curve of the tool, figuring out our server configurations for Dev, Int, and Prod, not to mention quickly putting together some form of standards with regards to parameters, directories, etc. ad nauseum.

We had Ken Bland out here a couple of months ago and we have been diligently trying to wrap our brains around all the expertice he imparted to us (mondo kudos to Ken by the way - we were very impressed by the training he provided as well as his willingness to tailor things to fit our project needs :lol: )

So, while we have spent time getting basics standardized, everyone has also been working on building jobs for testing, learning, and of course, the actual projects. We have finally gotten agreement on a base set of parameters and directory structures that will be common to all. The problem is that some industrious people (myself included) have already written hundreds of jobs with whatever parameters we thought we needed originally based off of some example jobs we got from one of our other companies that already had the tool and Ken's framework (another huge benefit to DataStage users everywhere!) If you are not familiar with the framework, search for it on the site. Learn it, implement it, live it! I strongly encourage anyone who is not using that methodology developed over the years by Ken, and others like Kim, Ray, Arndt, Craig, and countless others who I'm sure have contributed to it in some way or another, to contact some of these fine folks and invest in some of their time at your location. I promise it is well worth the investment and will pay for itself in no time at all :D -- (OK, I'm stepping off the soapbox now)

Now, we are all in the position of needing to put these new parameters in play without losing what we have already done. Oh yeah, to further complicate things, most of our development was done on a Windoze version of DataStage Server Edition. Now that we have our final servers, they are AIX Unix with completely different directory structures for our projects.

I know there are products out there that could help with this (such as Parameter Manager - I think that's the name). The problem is we needed to institute the changes and the move now, not in the month+ it will take to get corporate approval to purchase something. So, I took a couple of days and wrote two Perl scripts that will do the work for us (at least most of it).

I'm going to post them here in case anyone else can use them. I am not claiming to be a Perl maven, so I'm sure the scripts can be improved significantly. I wrote them with the goal of keeping them platform independent, so brevity is not always there. I have also avoided using system specific things such as backticks that could have turned entire sections of code into a single line or two. I did not use any obfuscated Perl in these scripts so they should be pretty easy to follow. I also tried to document as I went along. It should be pretty self explanatory.

Still, the speed is extremely good and appears to work for us so my hope is someone else can make use of it. Alternatively, if there are any super duper Perl'ers out there that can help make them better, I'm hoping you will share your expertice with the rest of us by modifying it and sending changes back for us all to benefit from.

Both scripts are non-destructive to the original file, creating instead secondary files that have the changes in them. Just read the documentation header at the start of each program and you should have all you need to use them. In fact, the documentation is longer than the actual programs by a fair piece. Hopefully, it is useful to you. Feel free to send questions/problems/revisions to the email in the file so I can improve them.

So, here is the first one, called 'ReplaceParameters.pl'. It is intended to take a parameter block the user isolates from an export of a template job into a file and then the Perl script will parse the export file, placing the block in all the appropriate places. It can handle an export of any size and processes very quickly. I tested with an export of over 200 jobs which generated a .dsx file that was 18.2 megabytes in size. It processed the file in just under six seconds on AIX Unix and around the same time in Windoze.

Code: Select all

#!/usr/local/bin/perl -w

###################################################################################################################
#
# Documentation Header
# 
###################################################################################################################
# Script Name: ReplaceParameters.pl
# Author     : John Miceli
# Create Date: 10/10/2006
#
# Purpose: This script is meant to quickly and easily replace parameter blocks in a DataStage export file (.dsx)
#          with the contents of file prepared in advance with the correct parameters.  This is a non-destructive
#          process that will not affect your original export file.  Instead, the process will create two additional
#          files (see Output section).
#
# Test results: I tested this script against an export file consisting of 212 extract jobs.  The file size was
#		just under 18.2 megabytes in size.  The script correctly processed the file in 6 seconds.
#
# Operating command: perl ReplaceParameters.pl filename.dsx parm_filename
# 		1. filename.dsx - this is the name of the export file (in .dsx export file)
# 		2. parm_filename - this is the name of the file that contains the correct parameters you are 
# 				    wishing to place in the export file
#
# Output     : 1. Original file (filename.dsx) - will remain untouched and is used for reading purposes only
# 	       2. changed_filename.dsx - will contain the original script minus the parameter blocks.  These 
# 	          			will be replaced by a place holder called 'PARAMETERS_GO_HERE'.
# 	       3. final_filename.dsx - will contain the finished product, ready to be imported
#
# Caveats    : I don't claim to be a Perl maven.  I'm sure someone smarter than me could have done this with
# 		fewer lines of code.  However, it works quickly and efficiently.  It is not complicated at all,
# 		hence the resulting performance.  Part of keeping it simple is that I may be missing some of the
# 		fluff found in lots of Perl code that probably should be there depending on your level of
# 		paranoia ;-)
#		
#		Assumptions made:
# 		I have assumed the user will include both parameters and that the correct filenames will be
# 		in the correct place.  If a filename is missing, then the script will simply puke.  Call
# 		me lazy, but I'm telling you how to use it, so just do it that way and don't do it any other;-).
# 		I have also assumed that the user is DataStage proficient enough to do the following (this 
# 		section also happens to be the instructions for using this script):
# 		1. Create a DS job that has nothing but the correct parameters defined
# 		2. Export the single parameter job to a .dsx file
# 		3. Use VI or editor of choice to remove everything above and below the parameter block 
# 		   (line 1 of the file down to and including the line 'Parameters "CParameters" ' for the top
# 		   part and everything from the line reading 'NULLIndicatorPosition "0" ' to the bottom of the
# 		   file).
# 		4. Save the leftover parameter block as some filename you can modify and use (I used 'parms').
# 		5. Export any number of jobs into an export file (.dsx format) that need the new parameters (I'll
# 		    refer to this as 'filename.dsx').
# 		6. Put this script into your path somewhere (if you haven't already done it) or into a working
# 		   directory you have access to.  Place the other two files in a working directory and issue the
# 		   following command (without the single quotes): 'perl ReplaceParameters.pl filename.dsx parms'
# 		   This will fire off the script.  When it is done, your new parameters will be in a .dsx file
# 		   prepended with 'final_' and will have your original filename tacked on the end 
# 		   (ie.: final_filename.dsx).
#
# 		!!!NOTE!!!   
# 		If there were any previously used parameters that had different names than this current parameter
# 		   scheme, you will need to fix these (they are referenced in different places in each stage).  
# 		   I have distributed with this script another called 'FixParameters.pl' which will allow 
# 		   you to replace parameters in the original file with new ones you manually map either one 
# 		   at a time or with a map file you will need to create.  See the header documentation in that file 
# 		   for the format of that mapping file.   
#
#	Any and all input/improvements to the script are welcome.  Either email me at 'jmiceli@wrberkley.com' or
#	message 'jdmiceli' on DSXchange.com and I will get back to you if I can.
#
#
###################################################################################################################
#
# Disclaimer: This script is provided AS IS and I take no responsibility for it's use or misuse.  It's use is completely
#       the responsibility of YOU the user and is at your own risk.  I do not claim perfection, nor do I warrantee or
#       guarantee the operation of the script.  It appears to work and I am using it and that's all the further I
#       am willing to go!!
#       	
###################################################################################################################
#
# Code Block
# 
###################################################################################################################

use strict;
use warnings;


#  define scalars
my $filename;  		# name of the file to be processed
my $parms;    		# holds name of parameters file
my $out_flag;  		# flag that will determine if something gets printed
my $changed_filename; 	# this will be the working file
my $final_file; 	# final file name for finished merged document 
my $line;      		# value for line for comparison

#
# collect the filename from the input argument and create the working filename
# 
$filename = "$ARGV[0]";
$parms = "$ARGV[1]";
$changed_filename = "changed_$filename";
$final_file = "final_$filename";

 open (OLD, "< $filename")||die "Unable to open $filename for reading!\n";
 open (NEW, "> $changed_filename")|| die "Unable to open $changed_filename for writing!\n";

#
# initialize out flag to normal mode: 0=normal line, 1=skip line
$out_flag = 0;

 while ($line = <OLD>)
 {
	chomp $line;

	#
	# see if the line is the beginning of a parameters section
	# 
	if ($line =~ /CParameters/)
	{
		print NEW "$line\n";
		$out_flag = 1;
	} 
	#
	# see if it is the end of the parameters section
	# 
	elsif ($line =~ /NULLIndicatorPosition/) 
	{
		print NEW "PARAMETERS_GO_HERE\n";
		$out_flag = 0;
	}

	#
	# final check
	#
	if ($out_flag == 0)
	{
		print NEW "$line\n";
	}
 }

close OLD;
close NEW;

#
# reopen the changed file and insert new parameters at 'PARAMETERS_GO_HERE' placeholder
# 
open (OLD, "< $changed_filename");
open (NEXT, "> $final_file");

while ($line = <OLD>)
{
	chomp $line;

	if ($line =~ /PARAMETERS_GO_HERE/)
	{
		open (CHANGED, "< $parms");

		while (<CHANGED>)
		{
			chomp;
			print NEXT "$_\n";
		}
		close CHANGED;
	} else
	{
		print NEXT "$line\n";
	}
}

close OLD;
close NEXT;


The second script is meant to be a follow up to the first. After replacing the parameter block, there may still be references to parameters or defaults from the original export. This script, called 'FixParameters.pl', will give you a mechanism to easily map the old values to the new values either one by one in a batch mode that will process a file you create that holds the pairs to be processed. The code will then create a new file prefacing your old filename with 'Fixed_' and step through the batch file you create and apply the changes to your .dsx file. Even though the original intent was for parameter and default replacement, the script really does not care. It will work on any file where you need to map changes. Bear in mind, it is pretty literal - so what you type for things to replace is what it will do. Don't use any extra stuff unless it is actually part of the old or new values. Examples are provided in the documentation header, so go there first. Send questions if you like and I will do my best to answer them or modify the program to do better.

Code: Select all

#!/usr/local/bin/perl -w

###################################################################################################################
#
# Documentation Header
# 
###################################################################################################################
# Script Name: FixParameters.pl
# Author     : John Miceli
# Create Date: 10/10/2006
#
# Purpose: This script is meant to easily replace parameters that have the wrong parameter name.
# 	    This will usually occur after using the Perl script 'ReplaceParameters.pl' to replace large blocks
# 	    of parameters in an export file from DataStage.  This script could also be used just to change 
# 	    default values for a set of parameters as opposed to just parameter names.  It's pretty flexible 
# 	    since it is just doing pattern matching and replacements.  
#
# 	    This script is non-destructive and will not affect your original file in any way.  A new file will
# 	    be created that prepends 'Fixed_' in front of the original filename.  Technically, this program
# 	    will work on any file, not just a .dsx file.  It really doesn't care.  I'm only using the export
# 	    for the example here because that is what I originally wrote the program for making mass changes
# 	    to export files for DS.  
#
# 	    The number of rows processed will print by 10,000 row counts, spitting out a final figure at 
# 	    the end.
#
# 	    NOTE!!!: I wrote this to be as platform independent as possible, so it won't care if you are on
# 	    Unix or Windows.  This is intentional and I realize by doing so, I passed on using Perl features
# 	    that could have simplified it greatly and probably sped things up tremendously by processing the 
# 	    entire file at once by using backticks and 'sed'.  However, 'sed' isn't available on all systems
# 	    (pronounced Windoze) so that type of pattern matching and replacing is not available without 3rd
# 	    party tools.  Maybe the version of the MKS Toolkit that comes with DataStage would be available 
# 	    for that use, but I'm ignorant of it's use.  Feel free to enlighten me and I'll rewrite this.
#
# Test results: I have run several tests with this script thus far on an export file with 716,844 lines in it.
# 		Running the program in 'I' mode processed the file in 8 seconds.  Running the program in 'B'
# 		mode with a batch_file containing 3 pairs to substitute took 11 seconds.  A batch_file containing
# 		32 complex pairs (directory defaults) processed the test export file in 36 seconds.  Your results may vary.
#
# Operating command: perl FixParameters.pl 'I' filename.dsx 'old_parm' 'new_parm' (Individual mode)
# 			or
# 		     perl FixParameters.pl 'B' filename.dsx batch_filename (Batch mode)
# 		     
# 		1. operation mode ('I' or 'B') - there are two modes of operation:
# 			Individual mode ('I') - allows you to pass in a pair of parameters, the first is the
# 						old parameter and the second is the new parameter.  The program
# 						will replace any instances of the old with the new
# 			Batch mode ('B') - allows you to prepare a file ahead of time that contains all the
# 					    'old=>new' parameter pairs to be processed.  The program will then 
# 					    load the file into a hash and process the pairs until finished.
# 		2. filename.dsx - this is the name of the export file (.dsx extension) to be processed
# 		3. batch_filename - this is the name of the file that contains pairs of 'old=>new' parameter
# 					values to be processed.  This file is prepared ahead of time and
# 					follows the same general format as insert values into a 
# 					hash (associative array).  The only difference is that you don't put 
# 					in the hash assignment as you would in Perl. 
# 					 
# 					An example would be as follows:
# 						"SourceUserID" => "SrcUserID"
# 						"SourceSystem" => "SrcDSN"
# 						"SourcePassword" => "SrcPwd"
# 					"E:\\datastage\\Dev\\LDR\\r1\\hash\\" => "//datastage//dev//ldr//r1//hash//"
#					"E:\\datastage\\Dev\\common\\r1\\hash\\" => "//datastage//dev//common//r1//hash//"

#		  Some rules to follow when creating the batch_file:
#		  	A. Type exactly as it will be found in a 'whole word'.  As above for directory structures 
#		  	   and if there are no quotes around the term, don't put them there.
#		  	B. The program is pretty literal.  It is smart enough to handle fixing the '\' or '/'
#		  	   characters for matching purposes, so you don't need to change anything in that respect.
#		  	   I'm certain there is something I've missed, so code will have to be changed for those
#		  	   odd pattern matching items that could be in a DS field.
#		  	C. One pair per line with no ending comma or punctuation.  
#		  	D. When typing in the ' => ' make sure it is exactly like that: one space, the =>, and one space.
#		  That's basically it for the rules for that file.
#		  
# Output     : 1. Original file (ie: filename.dsx) - will remain untouched and is used for reading purposes only
# 	       2. Fixed_filename.dsx - will contain the original script with the changed parameter names.
# 	       3. If the process ever breaks (nah, couldn't happen!), you may see a file called 'ff.tmp'.  This is 
# 	          a temp file that is used during the process, and then removed when the rename is done over the 
# 	          top of the Fixed_filename.
#
# Caveats    : I don't claim to be a Perl maven.  I'm sure someone smarter than me could have done this with
# 		fewer lines of code.  However, it works quickly and efficiently.  It is not complicated at all,
# 		hence the resulting performance.  Part of keeping it simple is that I may be missing some of the
# 		fluff found in lots of Perl code that probably should be there depending on your level of
# 		paranoia ;-)
#		
#		Assumptions made:
# 		I have assumed the user will include both parameters and that the correct filenames will be
# 		in the correct place.  If a filename is missing, then the script will simply puke.  Call
# 		me lazy, but I'm telling you how to use it, so just do it that way and don't do it any other ;-).
# 		I have also assumed that the user is DataStage proficient enough to do the following (this 
# 		section also happens to be the instructions for using this script):
# 		1. Once the 'ReplaceParameters.pl' script has been run to get the parameter block in the correct
# 		   places, then it is time to take care of individual problems.
# 		2. This script could also be used just to change default values for a set of parameters as
# 		   opposed to just parameter names.  It's pretty flexible and is just doing pattern matching
# 		   and replaces.
# 		3. Either prepare a file in advance with the pairs of 'old=>new' items to change or map the
# 		   few items you need to fix and run in your mode of choice. An example is given in Item 3 of
# 		   the Operating Command section above.  There is no limit to the number of pairs that
# 		   can be processed and the program doesn't really care if they are parameters or not.
# 		4. Run the program using the commands listed above.  You will find your changes in a file
# 		   that prepends "Fixed_" in front of the original filename.  This process will not affect the
# 		   original file in any way.  It is only used for non-destructive read.
#
#
#	Any and all input/improvements to the script are welcome.  Either email me at 'jmiceli@wrberkley.com' or
#	message 'jdmiceli' on DSXchange.com and I will get back to you if I can.
#
#
###################################################################################################################
#
# Disclaimer: This script is provided AS IS and I take no responsibility for it's use or misuse.  It's use is completely
#       the responsibility of YOU the user and is at your own risk.  I do not claim perfection, nor do I warrantee or
#       guarantee the operation of the script.  It appears to work and I am using it and that's all the further I
#       am willing to go!!
#       	
###################################################################################################################
#
# Code Block
# 
###################################################################################################################
use strict;
use warnings;

#  define scalars
my $filename;  		# name of the file to be processed
my $parms;    		# holds name of parameters file
my $fixed_filename; 	# this will be the working file
my $line;      		# value for line for comparison
my %parms;		# hash for holding the file contents for fixing
my $mode;		# mode for running. Values: 'I'=individual pairs, 'B'=batch processing using file for pairs
my $oldparm; 		# holds old value
my $newparm;		# holds new value

#
# collect the filename from the input argument and create the working filename
# 
$mode = "$ARGV[0]";
if ($mode eq "'I'")
{
	print "Mode is 'I'\n";
} 
elsif ($mode eq "'B'")
{
	print "Mode is 'B'\n";
} 
else
{
	die "Incorrect mode! Must be I or B in single quotes.\n";
}

$filename = "$ARGV[1]";
$fixed_filename = "Fixed_$filename";

if ($mode=~ /'I'/)
{
	$oldparm = "$ARGV[2]";
	$newparm = "$ARGV[3]";
	$parms{"$oldparm"} = "$newparm";
}

if ($mode=~ /'B'/)
{
	$parms = "$ARGV[2]";
}


#
# see if anything was passed into $parms and add the values to the $str scalar
# 
 
 if (defined ($parms))
 {
	open(PARMS, "< $parms");
	
	while ($line = <PARMS>)
	{
		chomp $line;
		
		($oldparm, $newparm) = split (/ => /, $line);
		#
		# check to see if there are slashes of either flavor in the scalar, then put in the escape for it
		#
		if ($oldparm =~ /\\/ || $oldparm=~ /\/\//)
		{
			$oldparm =~ s#//#////#g;
			$oldparm =~ s#\\#\\\\#g;
		}
		
		$parms{$oldparm} = $newparm;
		# print "Key: $oldparm, Value: $newparm\n";  # uncomment to see the output for debugging
	
	}
	close PARMS;	
}

#
# time to go through the input file and create the backup file
#

open(OLD, "<$filename");
open(NEW, ">$fixed_filename");

while ($line = <OLD>)
{
	chomp $line;
	print NEW "$line\n";
}

close OLD;
close NEW;

#
# make the mods to $fixed_filename
#
my $tmp = ("ff.tmp");
my $cnt = 0;

open(TMP, "> $tmp");
open(FIX, "< $fixed_filename");

while ($line = <FIX>)
{
	chomp $line;
	$cnt++;
	study $line;  				  # for longer comparisons and replaces, this should help

	while (($oldparm, $newparm) = each %parms)
	{
		$line =~ s/$oldparm/$newparm/g; # used different delimiters since slashes can be in the parms
		#print "Key: $oldparm, Value: $newparm\n";  # uncomment for debugging purposes
	}

	if ($cnt % 10000 == 0)
	{
		print "$cnt lines processed.\n";
	}
	
	print TMP "$line\n";
}

close TMP;
close FIX;

print "Total of $cnt lines processed.\n";

rename("$tmp", "$fixed_filename");

Once again, please feel free to drop me a line if you see something that needs fixed (very likely - so test carefully) or need clarification. Also, if you want me to just forward the files, including samples of the batch_file and/or parameter file, I'll be happy to do so on request.

Hope this helps!

Bestest,
Bestest!

John Miceli
System Specialist, MCP, MCDBA
Berkley Technology Services


"Good Morning. This is God. I will be handling all your problems today. I will not need your help. So have a great day!"
jdmiceli
Premium Member
Premium Member
Posts: 309
Joined: Wed Feb 22, 2006 10:03 am
Location: Urbandale, IA

oops!

Post by jdmiceli »

See! I told you I'd miss something! I usually run in batch mode so I forgot to include some pattern matching changes I made for dealing with directory structure replacement. The new version is here:

Code: Select all

#!/usr/local/bin/perl -w

###################################################################################################################
#
# Documentation Header
# 
###################################################################################################################
# Script Name: FixParameters.pl
# Author     : John Miceli
# Create Date: 10/10/2006
#
# Purpose: This script is meant to easily replace parameters that have the wrong parameter name.
# 	    This will usually occur after using the Perl script 'ReplaceParameters.pl' to replace large blocks
# 	    of parameters in an export file from DataStage.  This script could also be used just to change 
# 	    default values for a set of parameters as opposed to just parameter names.  It's pretty flexible 
# 	    since it is just doing pattern matching and replacements.  
#
# 	    This script is non-destructive and will not affect your original file in any way.  A new file will
# 	    be created that prepends 'Fixed_' in front of the original filename.  Technically, this program
# 	    will work on any file, not just a .dsx file.  It really doesn't care.  I'm only using the export
# 	    for the example here because that is what I originally wrote the program for making mass changes
# 	    to export files for DS.  
#
# 	    The number of rows processed will print by 10,000 row counts, spitting out a final figure at 
# 	    the end.
#
# 	    NOTE!!!: I wrote this to be as platform independent as possible, so it won't care if you are on
# 	    Unix or Windows.  This is intentional and I realize by doing so, I passed on using Perl features
# 	    that could have simplified it greatly and probably sped things up tremendously by processing the 
# 	    entire file at once by using backticks and 'sed'.  However, 'sed' isn't available on all systems
# 	    (pronounced Windoze) so that type of pattern matching and replacing is not available without 3rd
# 	    party tools.  Maybe the version of the MKS Toolkit that comes with DataStage would be available 
# 	    for that use, but I'm ignorant of it's use.  Feel free to enlighten me and I'll rewrite this.
#
# Test results: I have run several tests with this script thus far on an export file with 716,844 lines in it.
# 		Running the program in 'I' mode processed the file in 8 seconds.  Running the program in 'B'
# 		mode with a batch_file containing 3 pairs to substitute took 35 seconds.  A batch_file containing
# 		XX pairs processed the test export file in XX seconds.  Your results may vary.
#
# Operating command: perl FixParameters.pl 'I' filename.dsx 'old_parm' 'new_parm' (Individual mode)
# 			or
# 		     perl FixParameters.pl 'B' filename.dsx batch_filename (Batch mode)
# 		     
# 		1. operation mode ('I' or 'B') - there are two modes of operation:
# 			Individual mode ('I') - allows you to pass in a pair of parameters, the first is the
# 						old parameter and the second is the new parameter.  The program
# 						will replace any instances of the old with the new
# 			Batch mode ('B') - allows you to prepare a file ahead of time that contains all the
# 					    'old=>new' parameter pairs to be processed.  The program will then 
# 					    load the file into a hash and process the pairs until finished.
# 		2. filename.dsx - this is the name of the export file (.dsx extension) to be processed
# 		3. batch_filename - this is the name of the file that contains pairs of 'old=>new' parameter
# 					values to be processed.  This file is prepared ahead of time and
# 					follows the same general format as insert values into a 
# 					hash (associative array).  The only difference is that you don't put 
# 					in the hash assignment as you would in Perl. 
# 					 
# 					An example would be as follows:
# 						"SourceUserID" => "SrcUserID"
# 						"SourceSystem" => "SrcDSN"
# 						"SourcePassword" => "SrcPwd"
# 					"E:\\datastage\\Dev\\LDR\\r1\\hash\\" => "//datastage//dev//ldr//r1//hash//"
#					"E:\\datastage\\Dev\\common\\r1\\hash\\" => "//datastage//dev//common//r1//hash//"

#		  Some rules to follow when creating the batch_file:
#		  	A. Type exactly as it will be found in a 'whole word'.  As above for directory structures 
#		  	   and if there are no quotes around the term, don't put them there.
#		  	B. The program is pretty literal.  It is smart enough to handle fixing the '\' or '/'
#		  	   characters for matching purposes, so you don't need to change anything in that respect.
#		  	   I'm certain there is something I've missed, so code will have to be changed for those
#		  	   odd pattern matching items that could be in a DS field.
#		  	C. One pair per line with no ending comma or punctuation.  
#		  	D. When typing in the ' => ' make sure it is exactly like that: one space, the =>, and one space.
#		  That's basically it for the rules for that file.
#		  
# Output     : 1. Original file (ie: filename.dsx) - will remain untouched and is used for reading purposes only
# 	       2. Fixed_filename.dsx - will contain the original script with the changed parameter names.
# 	       3. If the process ever breaks (nah, couldn't happen!), you may see a file called 'ff.tmp'.  This is 
# 	          a temp file that is used during the process, and then removed when the rename is done over the 
# 	          top of the Fixed_filename.
#
# Caveats    : I don't claim to be a Perl maven.  I'm sure someone smarter than me could have done this with
# 		fewer lines of code.  However, it works quickly and efficiently.  It is not complicated at all,
# 		hence the resulting performance.  Part of keeping it simple is that I may be missing some of the
# 		fluff found in lots of Perl code that probably should be there depending on your level of
# 		paranoia ;-)
#		
#		Assumptions made:
# 		I have assumed the user will include both parameters and that the correct filenames will be
# 		in the correct place.  If a filename is missing, then the script will simply puke.  Call
# 		me lazy, but I'm telling you how to use it, so just do it that way and don't do it any other ;-).
# 		I have also assumed that the user is DataStage proficient enough to do the following (this 
# 		section also happens to be the instructions for using this script):
# 		1. Once the 'ReplaceParameters.pl' script has been run to get the parameter block in the correct
# 		   places, then it is time to take care of individual problems.
# 		2. This script could also be used just to change default values for a set of parameters as
# 		   opposed to just parameter names.  It's pretty flexible and is just doing pattern matching
# 		   and replaces.
# 		3. Either prepare a file in advance with the pairs of 'old=>new' items to change or map the
# 		   few items you need to fix and run in your mode of choice. An example is given in Item 3 of
# 		   the Operating Command section above.  There is no limit to the number of pairs that
# 		   can be processed and the program doesn't really care if they are parameters or not.
# 		4. Run the program using the commands listed above.  You will find your changes in a file
# 		   that prepends "Fixed_" in front of the original filename.  This process will not affect the
# 		   original file in any way.  It is only used for non-destructive read.
#
#
#	Any and all input/improvements to the script are welcome.  Either email me at 'jmiceli@wrberkley.com' or
#	message 'jdmiceli' on DSXchange.com and I will get back to you if I can.
#
#
###################################################################################################################
#
# Disclaimer: This script is provided AS IS and I take no responsibility for it's use or misuse.  It's use is completely
#       the responsibility of YOU the user and is at your own risk.  I do not claim perfection, nor do I warrantee or
#       guarantee the operation of the script.  It appears to work and I am using it and that's all the further I
#       am willing to go!!
#       	
###################################################################################################################
#
# Code Block
# 
###################################################################################################################
use strict;
use warnings;

#  define scalars
my $filename;  		# name of the file to be processed
my $parms;    		# holds name of parameters file
my $fixed_filename; 	# this will be the working file
my $line;      		# value for line for comparison
my %parms;		# hash for holding the file contents for fixing
my $mode;		# mode for running. Values: 'I'=individual pairs, 'B'=batch processing using file for pairs
my $oldparm; 		# holds old value
my $newparm;		# holds new value

#
# collect the filename from the input argument and create the working filename
# 
$mode = "$ARGV[0]";
if ($mode eq "'I'")
{
	print "Mode is 'I'\n";
} 
elsif ($mode eq "'B'")
{
	print "Mode is 'B'\n";
} 
else
{
	die "Incorrect mode! Must be I or B in single quotes.\n";
}

$filename = "$ARGV[1]";
$fixed_filename = "Fixed_$filename";

if ($mode=~ /'I'/)
{
	$oldparm = "$ARGV[2]";
	$newparm = "$ARGV[3]";
	#
	# check to see if there are slashes of either flavor in the scalar, then put in the escape for it
	#
	if ($oldparm =~ /\\/ || $oldparm=~ /\/\//)
	{
		$oldparm =~ s#//#////#g;
		$oldparm =~ s#\\#\\\\#g;
	}
	$parms{"$oldparm"} = "$newparm";
}

if ($mode=~ /'B'/)
{
	$parms = "$ARGV[2]";
}


#
# see if anything was passed into $parms and add the values to the $str scalar
# 
 
 if (defined ($parms))
 {
	open(PARMS, "< $parms");
	
	while ($line = <PARMS>)
	{
		chomp $line;
		
		($oldparm, $newparm) = split (/ => /, $line);
		#
		# check to see if there are slashes of either flavor in the scalar, then put in the escape for it
		#
		if ($oldparm =~ /\\/ || $oldparm=~ /\/\//)
		{
			$oldparm =~ s#//#////#g;
			$oldparm =~ s#\\#\\\\#g;
		}
		
		$parms{$oldparm} = $newparm;
		# print "Key: $oldparm, Value: $newparm\n";  # uncomment to see the output for debugging
	
	}
	close PARMS;	
}

#
# time to go through the input file and create the backup file
#

open(OLD, "<$filename");
open(NEW, ">$fixed_filename");

while ($line = <OLD>)
{
	chomp $line;
	print NEW "$line\n";
}

close OLD;
close NEW;

#
# make the mods to $fixed_filename
#
my $tmp = ("ff.tmp");
my $cnt = 0;

open(TMP, "> $tmp");
open(FIX, "< $fixed_filename");

while ($line = <FIX>)
{
	chomp $line;
	$cnt++;
	study $line;  				  # for longer comparisons and replaces, this should help

	while (($oldparm, $newparm) = each %parms)
	{
		$line =~ s/$oldparm/$newparm/g; # used different delimiters since slashes can be in the parms
		#print "Key: $oldparm, Value: $newparm\n";  # uncomment for debugging purposes
	}

	if ($cnt % 10000 == 0)
	{
		print "$cnt lines processed.\n";
	}
	
	print TMP "$line\n";
}

close TMP;
close FIX;

print "Total of $cnt lines processed.\n";

rename("$tmp", "$fixed_filename");

Bestest!
Bestest!

John Miceli
System Specialist, MCP, MCDBA
Berkley Technology Services


"Good Morning. This is God. I will be handling all your problems today. I will not need your help. So have a great day!"
kduke
Charter Member
Charter Member
Posts: 5227
Joined: Thu May 29, 2003 9:47 am
Location: Dallas, TX
Contact:

Post by kduke »

Good job. I do not like Perl. I think it is hard to read but you came up with a solution which works. Well done.

Most of us have used Universe for many years so we tend to deal with the repository tables directly with jobs or routines. This is a very clever solution. Thanks for sharing.
Mamu Kim
kduke
Charter Member
Charter Member
Posts: 5227
Joined: Thu May 29, 2003 9:47 am
Location: Dallas, TX
Contact:

Post by kduke »

By the way Ken's batch processes are totally his invention as far as I know. I don't think he has used any code from me or anyone else. I think Craig actually uses Ken's jobs and methodolgy.
Mamu Kim
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Well, actually... they are the 'KBA Job Control Utilities' and 'KBA' is Ken Bland and Associates. You'll see names like Stephen Wong, Michael Favero (and others I'm sure) in addition to Ken's.

And yah, been relying on it heavily for 3+ years now. I've extended the 'wrapper' code around it quite a bit, but at its heart it Pure KBA. Cool stuff. :wink:

John - you do know there is a DataStage job that come with it that does something very similar to your Perl code, but 'in place' right in the repository - yes? Was that your inspiration?

Batch::UTILSetDefaultParams

Part of a loverly little set of 'UTIL' jobs he throws in, the steak knifes of the Job Control offering. :lol:
-craig

"You can never have too many knives" -- Logan Nine Fingers
jdmiceli
Premium Member
Premium Member
Posts: 309
Joined: Wed Feb 22, 2006 10:03 am
Location: Urbandale, IA

There was???

Post by jdmiceli »

Hi Craig,

Actually, I had no idea whatsoever that there was something built in. Once again, my newbie ignorance of the tools shines forth. My main concern at the time was having a three day deadline for moving stuff from a Windoze DS environment to an AIX Unix environment. Not knowing about the 'Batch::UTILSetDefaultParams', I did what any 'normal' newbie does: I panicked :oops:

I'll take a look at the Batch::... utility and see what we can do with it.

Bestest!
Bestest!

John Miceli
System Specialist, MCP, MCDBA
Berkley Technology Services


"Good Morning. This is God. I will be handling all your problems today. I will not need your help. So have a great day!"
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Just to be clear for everyone else who might look for it, Batch::UTILSetDefaultParams comes from Ken Bland & Associates - it's not "built-in" to DataStage as jdmicelli suggests.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
jdmiceli
Premium Member
Premium Member
Posts: 309
Joined: Wed Feb 22, 2006 10:03 am
Location: Urbandale, IA

Post by jdmiceli »

Hi all,

I am going to go ahead and close this one out. However, the versions of the script here have already been debugged some more!

There are more current and proven versions of the two scripts available to you. If you would like it, just shoot me an email and I'll forward them to you. No sense posting here and jacking my points for no good reason.

Bestest!
Bestest!

John Miceli
System Specialist, MCP, MCDBA
Berkley Technology Services


"Good Morning. This is God. I will be handling all your problems today. I will not need your help. So have a great day!"
Post Reply