Sequence Job with Loop iterations in background.

ray.wurlod · Post by **ray.wurlod** » Mon May 26, 2014 3:08 pm

1. No. With a Loop, it goes around and around, one iteration at a time.

2. There is a final line terminator in your file. This means that you are generating an empty line when reading it. You need to eliminate that. Show how you convert your file to a delimited list, and we'll respond with that solution (usually in a User Variables activity).

joycerecacho · Post by **joycerecacho** » Mon May 26, 2014 3:21 pm

The Sequential File with the list has the following format:

Final delimiter ....... end
Delimiter .............. |
Quote ................... none

chulett · Post by **chulett** » Mon May 26, 2014 5:19 pm

To run a job in the loop without waiting, you can't use a Job Activity stage to run them as they have the wait built in. You'll need to use dsjob from the command line without either of the wait options to accomplish that.

ray.wurlod · Post by **ray.wurlod** » Mon May 26, 2014 7:02 pm

chulett wrote:To run a job in the loop without waiting, you can't use a Job Activity stage to run them as they have the wait built in. You'll need to use dsjob from the command line without either of the wait options to accomplish that.

That is, dsjob command without either -wait or -jobstatus options, invoked from an Execute Command activity.
It becomes your responsibility to monitor the executions, because you will have effectively disabled DataStage monitoring of job status by using this technique.
It also becomes your responsibility not to start so many instances of the job that your system resources are saturated.

chulett · Post by **chulett** » Mon May 26, 2014 7:59 pm

What he said.

chulett · Post by **chulett** » Tue May 27, 2014 7:48 am

You need to build them into the command line. Off the top of my head the syntax is -param parm=value for each name / value pair you need to pass in. So you would need three of them on the command line along with everything else. If that is what you tried and is not working for you, show us the exact syntax you are using to build the command line string.

joycerecacho · Post by **joycerecacho** » Tue May 27, 2014 9:00 am

Last trying:

/opt_dstage/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -param NOM_ARQ=`echo #StartLoop_Relatorio.$Counter#|cut -f1 -d'|'|cut -f1 -d'.'` CadastroPositivoTransacional CPS_Febraban_ValArquivo2

Result:

/opt_dstage/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -param NOM_ARQ=`echo ACPO005_60746948_11725176_20140501_001.60701190000104.5701|cut -f1 -d'|'|cut -f1 -d'.'` CadastroPositivoTransacional CPS_Febraban_ValArquivo2
Reply=255
Output from command ====>
Error running job

Status code = -2 DSJE_BADSTATE

In these examples I just informed one parameter to test but the correct is informing three.

chulett · Post by **chulett** » Tue May 27, 2014 9:13 am

Not a syntax error, that "bad state" error means it is not runnable - could mean not compiled but more than likely it is already running. You said earlier that it was a multi-instance job but I don't see any evidence of that in the command string, if it was you would need to pass the Invocation ID as part of the job name after a period: JobName.InvocationID

joycerecacho · Post by **joycerecacho** » Tue May 27, 2014 12:44 pm

You are right, Chulett. I havent informed the instance.
But even informing, another error happens:

Line command:

/opt_dstage/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -param NOM_ARQ=`echo #StartLoop_Relatorio.$Counter#|cut -f1 -d'.'` -param CNPJ_FONTE=`echo #StartLoop_Relatorio.$Counter#|cut -f2 -d'.'` -param COD_ARQ=`echo #StartLoop_Relatorio.$Counter#|cut -f3 -d'.'` CadastroPositivoTransacional CPS_Febraban_ValArquivo2.`echo #StartLoop_Relatorio.$Counter#|cut -f2 -d'.'`_`echo #StartLoop_Relatorio.$Counter#|cut -f3 -d'.'`

Result:

CPS_Febraban_ValArquivoDevDup2..JobControl (@Execute_Command_270): Command /opt_dstage/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -param NOM_ARQ=`echo did not finish OK, reply = '|cut -f1 -d'.'` -param CNPJ_FONTE=`echo '

It is about that third iteration that doesnt exist.

chulett · Post by **chulett** » Tue May 27, 2014 1:07 pm

Then it goes back to your "list of things" to iterate through - you have an extra delimiter at the end that you need to trim. That causes it to loop one last time with no valid values to parse.

joycerecacho · Post by **joycerecacho** » Tue May 27, 2014 1:13 pm

Look:

The 'cat' command that generates the parameters list has the following content:

ACPO005_60746948_11725176_20140501_001.60701190000104.5701|ACPO005_90400888_11725176_20140124_001.90400888000142.5700

There is no extra delimiter at the end (a third one).

First Iteration Result:
Output from command ====>

Status code = 0

Second Iteration Result:

Reply=|cut -f1 -d'.'` -param CNPJ_FONTE=`echo ACPO005_90400888_11725176_20140124_001.90400888000142.5700
Output from command ====>
|cut -f2 -d'.'` -param COD_ARQ=`echo ACPO005_90400888_11725176_20140124_001.90400888000142.5700

chulett · Post by **chulett** » Tue May 27, 2014 1:23 pm

What delimiter did you specify in the Start Loop stage? A pipe? Sorry, not sure what else to say other than the only time it goes back around again is when it hits a delimiter.

joycerecacho · Post by **joycerecacho** » Tue May 27, 2014 1:25 pm

Yeah, the delimiter is a pipe.

The parameter list has the following content:

ACPO005_60746948_11725176_20140501_001.60701190000104.5701|ACPO005_90400888_11725176_20140124_001.90400888000142.5700

Follow the log:

16:08:50: StartLoop_Relatorio loop iteration 1 started
16:08:50: Execute_Command_270 (COMMAND /opt_dstage/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -param NOM_ARQ=`echo ACPO005_60746948_11725176_20140501_001.60701190000104.5701|cut -f1 -d'.'` -param CNPJ_FONTE=`echo ACPO005_60746948_11725176_20140501_001.60701190000104.5701|cut -f2 -d'.'` -param COD_ARQ=`echo ACPO005_60746948_11725176_20140501_001.60701190000104.5701|cut -f3 -d'.'` CadastroPositivoTransacional CPS_Febraban_ValArquivo2.`echo ACPO005_60746948_11725176_20140501_001.60701190000104.5701|cut -f2 -d'.'`_`echo ACPO005_60746948_11725176_20140501_001.60701190000104.5701|cut -f3 -d'.'`) started
16:08:52: Execute_Command_270 finished, reply=0
16:08:52: StartLoop_Relatorio loop iteration 2 started
16:08:52: Execute_Command_270 (COMMAND /opt_dstage/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -param NOM_ARQ=`echo ACPO005_90400888_11725176_20140124_001.90400888000142.5700) started
16:08:52: Execute_Command_270 finished, reply=-1
16:08:52: Exception raised: @Execute_Command_270, Unhandled failure (-1) encountered executing command /opt_dstage/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -param NOM_ARQ=`echo ACPO005_90400888_11725176_20140124_001.90400888000142.5700

The first iteration finished OK, the second one was not called.

There is no third iteration anymore, however it aborts at the last (the second).

joycerecacho · Post by **joycerecacho** » Tue May 27, 2014 2:49 pm

Now I tested with 3 iterations.
(now the loop parameter list has 3 values)

It is interesting cause always at the last iteration the command is truncated. That problem which used to run one more iteration doesnt happen anymore cause I cut the last delimiter. Yeah, there was a delimiter.

If I run the job with just one iteration the command is also truncated, and always at the same point as follow:

/opt_dstage/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -param NOM_ARQ=`echo ACPO005_90400888_11725176_20140124_002.90400888000142.5711[...]

If I run it with a list of 3 iterations, this command truncation happens with the third iteration only. If I run 2 iterations, it happens with the second only, and if I run the job with just one iteration, it happens with this one.

It truncates the command at the same point, exactly at the same point.

I couldnt figure it out yet.

ssnegi · Post by **ssnegi** » Wed May 28, 2014 2:32 am

Try changing the parameter list delimiter to comma instead of pipe. It could be that having the same delimiter as the command is having an adverse effect on the job.