Looping using Sequencers in a sequence job?

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Looping using Sequencers in a sequence job?

Post by jackson.eyton »

Hi guys,
I have devised what I thought was a method of running an infinite loop in a sequence job. Basically a DO WHILE. I will explain the process in a moment but I went for this method as opposed to using the Loop Activity stages because I wanted the loop to exit on a condition, which those activities did not appear they would do.

My job layout is as follows:

Code: Select all

                              SLP1                             SLP2             RUN_JOB
                            /        ^                        /         ^        ^
                          v           \                      v             \      /

WaitFile - - - SEQ1               NC1 - - - SEQ2               NC2

                          \            ^                     \             ^     \
                           v         /                         v          /         v
                             CMD1                            CMD2             EMAIL

Sorry for the weird chemical formula looking diagram, my employer does not allow screenshots of anything internal to uploaded. Basically the job is designed to do the following:
Wait for a trigger file, then go to the first sequencer, SEQ1, set to ANY. Flow then continues to CMD1 which checks the status of a job in a different project (dsjob command). This output is sent over the the first Nested Condition, NC1, which runs an index() on the output to see if the job is running or not. If NOT then sends the flow to a sleep command, SLP1, which waits for a time and sends flow back to SEQ1 to repeat until the job being checked is in a running status. At which point flow then hits the second sequencer, SEQ2, set to ANY, and drops down to it's command, CMD2. This checks the same job status again and flows to the second Nested Condition, NC2. Here the job status is checked to see if the job completed, OK or with warnings, or is still running, or is not in any of those statuses. If it's still running SLP2, sleep, will be triggered and flow should go back around to check the status again until the job is completed or failed (no longer running nor completed). If completed, NC2 will trigger the link to run a job. If not completed and not running:

Code: Select all

Index(CheckJobStatus2.$CommandOutput, 'Job Status	: RUNNING (0)',1) = 0 Or Index(CheckJobStatus2.$CommandOutput, 'Job Status	: RUN with WARNINGS (2)',1) = 0 Or Index(CheckJobStatus2.$CommandOutput, 'Job Status	: RUN OK (1)',1) = 0
Then an email will be sent. My issue is that when I tested this I got both a FAILED link to run AND the Still Running link. The log then showed the following message (not error):
RawStageTrigger..JobControl (@Coordinator): Note: Sequencer 'SEQ2' was entered, but never exited

Any ideas why?
Last edited by jackson.eyton on Mon Oct 29, 2018 12:23 pm, edited 2 times in total.
-Me
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Upon further review, sometimes typing things out, reading it, then reading it again in a few minutes helps.... I see that my Or conditions in the code example I gave, should be And conditions....

Code: Select all

Index(CheckJobStatus2.$CommandOutput, 'Job Status	: RUNNING (0)',1) = 0 And Index(CheckJobStatus2.$CommandOutput, 'Job Status	: RUN with WARNINGS (2)',1) = 0 And Index(CheckJobStatus2.$CommandOutput, 'Job Status	: RUN OK (1)',1) = 0
-Me
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Ha! No-one has ever had that problem before. :wink:
-craig

"You can never have too many knives" -- Logan Nine Fingers
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Ok, so tested this again and confirmed it still doesn't work. It appears the job shuts down when the flow goes back to a sequencer that was not exited. Anyone know a functional way to run an infinite loop in a sequence job? DO/WHILE?
-Me
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

"Note: Sequencer 'SEQ2' was entered, but never exited"

that means you have it set to "All" and not all input links were triggered. Maybe it needs to be "Any"? Sorry, haven't taken the time to fully absorb your design / requirements but thought that might help. And on the "infinite loop" question I really don't think that's possible. :?
-craig

"You can never have too many knives" -- Logan Nine Fingers
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

The two sequences used are definitely set to ANY, datastage just doesn't seem to like running a loop back to the same sequence. Essentially it bombs when the process flow follows a path as follows:

Sequence Stage, Exec Command, Nested Condition, Exec Command, back to Sequence Stage. This is laid out in a circle.

Ok, I went around the rules a bit here and got the image uploaded here:
https://ibb.co/iOzRHf
-Me
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Hmmm... from what I recall they don't like "circles" either but there's a specific fatal error for that, I thought. But maybe that's the root of your issue. :?
-craig

"You can never have too many knives" -- Logan Nine Fingers
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Okay, just looked at your canvas. The "circles" I meant were when the end loops back to the beginning, I have vague memories of trying to do something like that back in the day and getting busted for it. But still, there's no way even those smaller sequencer circles will work like that. Perhaps as a Start/End Loop construct? Easy enough to fly around in one if conditions aren't met and then exit past the End Loop to move forward when they are. From what I recall. :wink:
-craig

"You can never have too many knives" -- Logan Nine Fingers
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

Seems like I ran into that circular error once myself and just moved on without trying to debug it.

We use begin/end loops with parameters for the amount of sleep time and number of retries. While it's not an infinite loop, you could set the number of retries to a very high number if that would help.

We set these up at the highest level of our sequence jobs and place that job on schedule. We did that in order to auto-retry failed jobs, because most of the time our jobs fail in the middle of the night is because of a temporary network glitch. A simple automatic retry (when jobs are set to the action of reset then run) is able to work around the shoddy network in most cases.
Choose a job you love, and you will never have to work a day in your life. - Confucius
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Ok, using the loop stages was my initial plan but I figured I would try doing an infinite. Not a huge deal, set the number of steps to 9999 with 5 second wait times that still 13 hours a loop could run. MORE than enough time. I do have an issue however..... the first loop setup fine and should exit to another loop. However, the second loop will not allow me to return link on the End Loop stage back to the Start Loop stage for that loop. Please see image here:
https://ibb.co/n3qbk0

If I remove the "Running" link from StartLoop_Activity_155 I can then link the End Loop to the start but it's a standard link, not the dotted line link (sorry the name escapes me)
-Me
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

That's odd and sounds like a bug in Designer. The same job design works fine on my Designer v11.3.1.2. What's the exact version you are on? We ran into something similar one time on a dot-zero release and had to get a patch from Support.
Choose a job you love, and you will never have to work a day in your life. - Confucius
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Hmmm wonder if that is it... I was able to trick it to giving me a link from the end loop back to the start loop by removing the end loop stage, adding a new one with no incoming links to it. Linking that to the start loop, which correctly showed the dotted line link. Then re-linking things to the end loop. So the designer shows all the links are there correctly. However, when I compile it still says the start loop has no end loop that links back to it.... We're on 11.5.0.1. I may open a case with IBM on it....
-Me
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

IBM has confirmed the job compiles on their end using 11.5.0.2, working on getting patches installed.
-Me
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Marking this as Resolved since IBM confirmed an issue with rollup version in our environment. No idea when it'll get fixed here...
-Me
Post Reply