DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 141

Points: 2315

Post Posted: Mon Oct 29, 2018 12:11 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Server
OS: Windows
Hi guys,
I have devised what I thought was a method of running an infinite loop in a sequence job. Basically a DO WHILE. I will explain the process in a moment but I went for this method as opposed to using the Loop Activity stages because I wanted the loop to exit on a condition, which those activities did not appear they would do.

My job layout is as follows:

Code:

                              SLP1                             SLP2             RUN_JOB
                            /        ^                        /         ^        ^
                          v           \                      v             \      /

WaitFile - - - SEQ1               NC1 - - - SEQ2               NC2

                          \            ^                     \             ^     \
                           v         /                         v          /         v
                             CMD1                            CMD2             EMAIL



Sorry for the weird chemical formula looking diagram, my employer does not allow screenshots of anything internal to uploaded. Basically the job is designed to do the following:
Wait for a trigger file, then go to the first sequencer, SEQ1, set to ANY. Flow then continues to CMD1 which checks the status of a job in a different project (dsjob command). This output is sent over the the first Nested Condition, NC1, which runs an index() on the output to see if the job is running or not. If NOT then sends the flow to a sleep command, SLP1, which waits for a time and sends flow back to SEQ1 to repeat until the job being checked is in a running status. At which point flow then hits the second sequencer, SEQ2, set to ANY, and drops down to it's command, CMD2. This checks the same job status again and flows to the second Nested Condition, NC2. Here the job status is checked to see if the job completed, OK or with warnings, or is still running, or is not in any of those statuses. If it's still running SLP2, sleep, will be triggered and flow should go back around to check the status again until the job is completed or failed (no longer running nor completed). If completed, NC2 will trigger the link to run a job. If not completed and not running:
Code:
Index(CheckJobStatus2.$CommandOutput, 'Job Status   : RUNNING (0)',1) = 0 Or Index(CheckJobStatus2.$CommandOutput, 'Job Status   : RUN with WARNINGS (2)',1) = 0 Or Index(CheckJobStatus2.$CommandOutput, 'Job Status   : RUN OK (1)',1) = 0


Then an email will be sent. My issue is that when I tested this I got both a FAILED link to run AND the Still Running link. The log then showed the following message (not error):
RawStageTrigger..JobControl (@Coordinator): Note: Sequencer 'SEQ2' was entered, but never exited

Any ideas why?

_________________
-Me

Last edited by jackson.eyton on Mon Oct 29, 2018 12:23 pm; edited 2 times in total
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 141

Points: 2315

Post Posted: Mon Oct 29, 2018 12:13 pm Reply with quote    Back to top    

Upon further review, sometimes typing things out, reading it, then reading it again in a few minutes helps.... I see that my Or conditions in the code example I gave, should be And conditions....

Code:
Index(CheckJobStatus2.$CommandOutput, 'Job Status   : RUNNING (0)',1) = 0 And Index(CheckJobStatus2.$CommandOutput, 'Job Status   : RUN with WARNINGS (2)',1) = 0 And Index(CheckJobStatus2.$CommandOutput, 'Job Status   : RUN OK (1)',1) = 0

_________________
-Me
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43035
Location: Denver, CO
Points: 222143

Post Posted: Mon Oct 29, 2018 1:33 pm Reply with quote    Back to top    

Ha! No-one has ever had that problem before. Wink

_________________
-craig

I've all my wisdom teeth, Two up top, two beneath
And yet I'll recognize my mouth says things that aren't so wise
Rate this response:  
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 141

Points: 2315

Post Posted: Tue Oct 30, 2018 5:35 am Reply with quote    Back to top    

Ok, so tested this again and confirmed it still doesn't work. It appears the job shuts down when the flow goes back to a sequencer that was not exited. Anyone know a functional way to run an infinite loop in a sequence job? DO/WHILE?

_________________
-Me
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43035
Location: Denver, CO
Points: 222143

Post Posted: Tue Oct 30, 2018 5:55 am Reply with quote    Back to top    

"Note: Sequencer 'SEQ2' was entered, but never exited"

that means you have it set to "All" and not all input links were triggered. Maybe it needs to be "Any"? Sorry, haven't taken the time to fully absorb your design / requirements but thought that might help. And on the "infinite loop" question I really don't think that's possible. Confused

_________________
-craig

I've all my wisdom teeth, Two up top, two beneath
And yet I'll recognize my mouth says things that aren't so wise
Rate this response:  
Not yet rated
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 141

Points: 2315

Post Posted: Tue Oct 30, 2018 8:34 am Reply with quote    Back to top    

The two sequences used are definitely set to ANY, datastage just doesn't seem to like running a loop back to the same sequence. Essentially it bombs when the process flow follows a path as follows:

Sequence Stage, Exec Command, Nested Condition, Exec Command, back to Sequence Stage. This is laid out in a circle.

Ok, I went around the rules a bit here and got the image uploaded here:
https://ibb.co/iOzRHf

_________________
-Me
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43035
Location: Denver, CO
Points: 222143

Post Posted: Tue Oct 30, 2018 9:27 am Reply with quote    Back to top    

Hmmm... from what I recall they don't like "circles" either but there's a specific fatal error for that, I thought. But maybe that's the root of your issue. Confused

_________________
-craig

I've all my wisdom teeth, Two up top, two beneath
And yet I'll recognize my mouth says things that aren't so wise
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43035
Location: Denver, CO
Points: 222143

Post Posted: Tue Oct 30, 2018 9:36 am Reply with quote    Back to top    

Okay, just looked at your canvas. The "circles" I meant were when the end loops back to the beginning, I have vague memories of trying to do something like that back in the day and getting busted for it. But still, there's no way even those smaller sequencer circles will work like that. Perhaps as a Start/End Loop construct? Easy enough to fly around in one if conditions aren't met and then exit past the End Loop to move forward when they are. From what I recall. Wink

_________________
-craig

I've all my wisdom teeth, Two up top, two beneath
And yet I'll recognize my mouth says things that aren't so wise
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2877
Location: USA
Points: 21833

Post Posted: Wed Oct 31, 2018 5:14 am Reply with quote    Back to top    

Seems like I ran into that circular error once myself and just moved on without trying to debug it.

We use begin/end loops with parameters for the amount of sleep time and number of retries. While it's not an infinite loop, you could set the number of retries to a very high number if that would help.

We set these up at the highest level of our sequence jobs and place that job on schedule. We did that in order to auto-retry failed jobs, because most of the time our jobs fail in the middle of the night is because of a temporary network glitch. A simple automatic retry (when jobs are set to the action of reset then run) is able to work around the shoddy network in most cases.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 141

Points: 2315

Post Posted: Wed Oct 31, 2018 7:13 am Reply with quote    Back to top    

Ok, using the loop stages was my initial plan but I figured I would try doing an infinite. Not a huge deal, set the number of steps to 9999 with 5 second wait times that still 13 hours a loop could run. MORE than enough time. I do have an issue however..... the first loop setup fine and should exit to another loop. However, the second loop will not allow me to return link on the End Loop stage back to the Start Loop stage for that loop. Please see image here:
https://ibb.co/n3qbk0

If I remove the "Running" link from StartLoop_Activity_155 I can then link the End Loop to the start but it's a standard link, not the dotted line link (sorry the name escapes me)

_________________
-Me
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2877
Location: USA
Points: 21833

Post Posted: Wed Oct 31, 2018 10:07 am Reply with quote    Back to top    

That's odd and sounds like a bug in Designer. The same job design works fine on my Designer v11.3.1.2. What's the exact version you are on? We ran into something similar one time on a dot-zero release and had to get a patch from Support.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 141

Points: 2315

Post Posted: Fri Nov 02, 2018 12:59 pm Reply with quote    Back to top    

Hmmm wonder if that is it... I was able to trick it to giving me a link from the end loop back to the start loop by removing the end loop stage, adding a new one with no incoming links to it. Linking that to the start loop, which correctly showed the dotted line link. Then re-linking things to the end loop. So the designer shows all the links are there correctly. However, when I compile it still says the start loop has no end loop that links back to it.... We're on 11.5.0.1. I may open a case with IBM on it....

_________________
-Me
Rate this response:  
Not yet rated
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 141

Points: 2315

Post Posted: Thu Nov 08, 2018 3:20 pm Reply with quote    Back to top    

IBM has confirmed the job compiles on their end using 11.5.0.2, working on getting patches installed.

_________________
-Me
Rate this response:  
Not yet rated
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 141

Points: 2315

Post Posted: Mon Nov 19, 2018 11:15 am Reply with quote    Back to top    

Marking this as Resolved since IBM confirmed an issue with rollup version in our environment. No idea when it'll get fixed here...

_________________
-Me
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours