DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
atulgoel
Participant



Joined: 03 Feb 2009
Posts: 83
Location: Bangalore, India
Points: 925

Post Posted: Wed Sep 04, 2019 8:08 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
I am running a Multi Instance Datastage in a loop by passing different parameters at run time sequencially using Unix Shell Script and DSJOB -run command. After running 4 to 5 iterations its gets out of loop and aborts. Does any one has any idea, is it related to some datastage resource issue.

Below is the output of Unix shell script and commands that I am using in Unix:

==================================================================
Script:

echo "Making entry in Audit table for below tables - CRZ"

echo "`grep "Loading data to table" $LocalAvscPath"controlfiles/bb5gsg_CRZ_Logs_"$SubGroup$Country.txt | cut -d " " -f8 | cut -d "_" -f6-7 --output-delimiter='|'`"
echo ""
echo "Starting Loop"
echo ""
for i in `grep "Loading data to table" $LocalAvscPath"controlfiles/bb5gsg_CRZ_Logs_"$SubGroup$Country.txt | cut -d " " -f8 | cut -d "_" -f6-7 --output-delimiter='|' |sort | uniq`
do

TABLANAME=`echo $i | cut -d'|' -f 1`
Version=`echo $i | cut -d'|' -f 2`

echo $TABLANAME
echo $Version

. /opt/IBM/InformationServer/Server/DSEngine/dsenv > /dev/null 2>&1

echo "/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=$Params -param ParamsHive_bb5gsg=$Params_Hive -param zone=$zone -param SubGroup=$SubGroup -param Country=$Country -param tablenm=$TABLANAME -param schemaVersion=$Version -param businessEffectiveDate=$businessEffectiveDate -param status=$status -param RecordCount=$RecordCount -param stream=$stream $Project $Job.$Group$SubGroup$CC$zone$TABLANAME"

/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=$Params -param ParamsHive_bb5gsg=$Params_Hive -param zone=$zone -param SubGroup=$SubGroup -param Country=$Country -param tablenm=$TABLANAME -param schemaVersion=$Version -param businessEffectiveDate=$businessEffectiveDate -param status=$status -param RecordCount=$RecordCount -param stream=$stream $Project $Job.$Group$SubGroup$CC$zone$TABLANAME > /dev/null 2>&1

RETURNCODE=$?

if [ $RETURNCODE = 1 -o $RETURNCODE = 2 ]
then
echo "Job completed successfully"
echo ""
else
echo "Job aborting with Return Code :"$RETURNCODE
exit 3
fi;
sleep 3
done
echo "Entry in Audit table Finished - CRZ"
echo ""

=================================================================
Output of UNIX Shell Script:

Making entry in Audit table for below tables - TSZ
achbatch|2
adt495|1
fpm425|2
fpt010|2
fpt041|2
fpt043|2
fpx120|3
fpx362|2
fpx391|2
gpt340|2
ipt471|2
opm090|2
rpm090|4


Starting Loop

achbatch
2
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=achbatch -param schemaVersion=2 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszachbatch
Job completed successfully

adt495
1
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=adt495 -param schemaVersion=1 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszadt495
Job completed successfully

fpm425
2
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=fpm425 -param schemaVersion=2 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszfpm425
Job completed successfully

fpt010
2
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=fpt010 -param schemaVersion=2 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszfpt010
Job completed successfully

fpt041
2
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=fpt041 -param schemaVersion=2 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszfpt041
Job completed successfully

fpt043
2
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=fpt043 -param schemaVersion=2 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszfpt043
Job aborting with Return Code :255

_________________
Atul
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1283

Points: 8443

Post Posted: Wed Sep 04, 2019 5:29 pm Reply with quote    Back to top    

Ask your datastage admin to look at the log purging setting in your project.

Under the Administrator tool, have him log in and tell you how the logs are purged.

There are two different settings. 1 is age based. Deleting log files or entries older than X amount of days.

The other is iteration based. Deleted log entries from jobs after N amount of runs. My guess is that this is what you are running into.

These log retention policies are set at a project level, not at the job level.
Rate this response:  
Not yet rated
atulgoel
Participant



Joined: 03 Feb 2009
Posts: 83
Location: Bangalore, India
Points: 925

Post Posted: Wed Sep 04, 2019 8:42 pm Reply with quote    Back to top    

Thanks for the reply. What should be the ideal setting for purging in order to resolve the issue?

_________________
Atul
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1283

Points: 8443

Post Posted: Thu Sep 05, 2019 1:51 pm Reply with quote    Back to top    

Your admin will make a recommendation after he talks to your application team and understands their log needs.

IF this is the issue of course.
Rate this response:  
Not yet rated
JRodriguez



Group memberships:
Premium Members

Joined: 19 Nov 2005
Posts: 421
Location: New York City
Points: 4714

Post Posted: Sat Sep 07, 2019 9:25 pm Reply with quote    Back to top    

How long it takes since the starting of the execution until it failed? It might be that the inactivity time out setting in DataStage Administrator is too low and the client connection that is running the job is closed... if this is the case then augmenting the value for the time out activity will do the trick

What happen if you take the command that is running the last job and run it manually, out of the loop, from the command line?

_________________
Julio Rodriguez
ETL Developer by choice

"Sure we have lots of reasons for being rude - But no excuses
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours