DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 106

Points: 1791

Post Posted: Tue Dec 26, 2017 10:04 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Server
OS: Windows
So this is likely a dumb question but I am thus far unable to find an answer to this. I have a sequence job that needs to call another sequence job and then end. The bet way I could get this to call the other job, and then go ahead and end, and not wait for the other job to complete, is to use dsjob.exe to call the other job. The arguments for this require the project name and job name of course. In the process of developing this and testing, I will need to package and move the jobs for my specific task from Dev to Test, and then from Test to Prod. These are all different projects and in the case of Prod, a different server set entirely. I'm hoping that the project name where the job resides is available as an environment variable so that I don't have to go and change the Execute Command stage arguments for each move. Something like $DSJProj maybe? Obviously not a HUGE issue more of a convenience.

_________________
-Me
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1237

Points: 8121

Post Posted: Tue Dec 26, 2017 2:44 pm Reply with quote    Back to top    

I would never execute a dsjob command to run a job from a first job. Your first job will end and give a return code to the calling script. The second job will run but have nobody to notify upon failure or success. danger danger.

Not sure why you would EVER want to fire off a job in prod without return code validation.
Rate this response:  
Not yet rated
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 106

Points: 1791

Post Posted: Tue Dec 26, 2017 4:21 pm Reply with quote    Back to top    

Well, the jobs being executed are sequence jobs with exception handling, so I would assume that would still work and I would get the designed email in case of failure. Additionally I can't think of a better way to do what I need. The scenario is as follows. The sequence jobs are replicas of each other. Once the sequence is started(running) it waits for a file to appear in a certain directory. This file can come at any time as its submitted to us by one of our customers, multiple files could be submitted throughout any given time period. Once the file is there the sequence runs a series of parallel jobs and server commands that read the contents of the file and store that in the database, then move the file to a processed folder with the datetime appended to the file name. The task then needs to repeat as "on-hold" for the next file, whenever it may come. I tried several ways to do this last part, first I tried simply calling a job activity stage to run itself again, but this seemed to cause issues. Second, I created the copy sequence job and had each one using the Job Activity stage at the end to call the other, but this caused a loop, in that whichever job was run first would never actually finish because it was waiting for the other sequence to complete, which would in turn also never finish as it was waiting for the first to finish, and so on. So this third option uses Execute Command at the end to "run" the sequence copy of the other at the end so in this manner the sequence jobs will always go back and forth waiting for the next file to process but will indeed successfully END once the copy job is called. I am more than open to ideas on this if there is a better way to approach it?

I suppose I could add an additional Notification Activity at the very end of the sequence that only emails if the previous Execute Command stage fails its DSJOB command...

_________________
-Me
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42576
Location: Denver, CO
Points: 219102

Post Posted: Tue Dec 26, 2017 6:43 pm Reply with quote    Back to top    

To address the original question, I don't recall the current project name being stored anywhere, i.e. a macro or system variable or anything of that nature... but it has been awhile. You can get a list of all of the projects through the API but I think that if you need the current one you'll need to create a user-defined environment variable in each project with that value.

_________________
-craig

I know I don't say this enough, but I like when you talk to me. It's much better than when nobody talks to me. Or when people that I don't like will not stop talking to me.
Rate this response:  
Not yet rated
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 679
Location: Malvern, PA
Points: 6425

Post Posted: Wed Dec 27, 2017 8:16 am Reply with quote    Back to top    

Sounds like a design problem (ahem, call me Dr. Obvious) but not at the DataStage level. You need a scheduler with cyclic capabilities to control your job runs.

If you don't have an external scheduler, the following might help you find an inconvenient alternative in DS.

We use Control-M. I would put the file watcher in the CM process, which would invoke your DS process each time a file came in, accept the return code, and either throw a failure alert or reset for the next file. You would need to find something to put in the invocation id to create distinct logs in Director, or relax the line limit on jobs because there'd be dozens of messages for a single job running multiple times.

Schedulers are just too useful to ignore. They give you an enhanced flexibility while taking on workload (and design efforts) that won't be easy or efficient in DS.

EDIT: re project name. If you use a ssi file in your dsjob script, create a project name variable there. You should have a naming convention standard for project names, something with a pattern to it. We use three parts to each name: Unix group name, project app prefix, syslevel. Ex. ABC_XYZ_PRD. Those also being parameters for the dsjob command line keeps us honest. Wink

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596 Using CFF FAQ: http://www.dsxchange.com/viewtopic.php?t=157872
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1237

Points: 8121

Post Posted: Wed Dec 27, 2017 9:59 am Reply with quote    Back to top    

So how about a loop in your master sequencer?

Test for file, if not present sleep 5 mins, loop to start. If present, continue to execute jobs, loop back to start when done...
Rate this response:  
Not yet rated
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 679
Location: Malvern, PA
Points: 6425

Post Posted: Wed Dec 27, 2017 10:06 am Reply with quote    Back to top    

PaulVL wrote:
So how about a loop in your master sequencer?


How would he exit the loop? It's an assumption, but he indicates that the job is time based.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596 Using CFF FAQ: http://www.dsxchange.com/viewtopic.php?t=157872
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42576
Location: Denver, CO
Points: 219102

Post Posted: Wed Dec 27, 2017 12:49 pm Reply with quote    Back to top    

FranklinE wrote:
EDIT: re project name. If you use a ssi file in your dsjob script, create a project name variable there.

Sorry... "ssi file"?

_________________
-craig

I know I don't say this enough, but I like when you talk to me. It's much better than when nobody talks to me. Or when people that I don't like will not stop talking to me.
Rate this response:  
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 679
Location: Malvern, PA
Points: 6425

Post Posted: Wed Dec 27, 2017 12:58 pm Reply with quote    Back to top    

"Server side includes." The general use is to source in script lines that are common to scripts in use. In my experience, they are common function definitions -- get server name, verify existence of directories, etc. -- and in our DataStage environment they contain required common variables and parameters at the server level. We customize them to include the environment variables, and use a utility script to update the project when we make changes to them.

Example lines from a project ssi file, with names changed to protect the innocent:
Code:
G_DS_ENV=(syslevel)
G_DS_UNIX_GROUP=abc
ENV_CAPS=$(echo ${G_DS_ENV} | tr [a-z] [A-Z] )
GRP_CAPS=$(echo ${G_DS_UNIX_GROUP} | tr [a-z] [A-Z] )
G_DS_PRJ=XYZ

G_DS_PROJ_NAME=${GRP_CAPS}_${G_DS_PRJ}_${ENV_CAPS}

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596 Using CFF FAQ: http://www.dsxchange.com/viewtopic.php?t=157872
Rate this response:  
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 106

Points: 1791

Post Posted: Wed Dec 27, 2017 1:12 pm Reply with quote    Back to top    

Forgive my ignorance, I am not familiar with any schedulers as you have referred to Sad I am pretty new to InfoSphere and ETL in general. If you could elaborate on the dangers of using DSJOB to execute another job? When doing so I can see the job running in the Director and review the logs from that job. Also the notification Activity stages in the jobs configured will let me know if there's ever a problem processing, so I'm having a hard time understanding what exactly the drawback is, I do apologize...

_________________
-Me
Rate this response:  
Not yet rated
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 679
Location: Malvern, PA
Points: 6425

Post Posted: Wed Dec 27, 2017 1:31 pm Reply with quote    Back to top    

Told to me by my Cobol instructor: never apologize for ignorance. It gives those of us with a little bit of knowledge a chance to look like geniuses. Laughing

DataStage has a scheduling utility. Look it up in the manuals. It's rather primitive compared to external schedulers, and might suit you if your shop doesn't have or won't get one.

In the case of Control-M, it's a job which has calendar and time attributes, and facilitates building cycles of jobs which you can control in various ways. It's used in many larger shops with mainframes and a long list of jobs.

The CM job's core is a command line. In my case, it invokes a script which builds and submits a dsjob command line. It has error handling and notifications built in, and in our large shop is monitored by our Ops crew. When abends happen, they go to documentation for what to do or whom to call.

In short, it covers all of the tasks you have to build into the DS jobs, in a standardized fashion which removes the necessity of doing it in every individual DS job.

It's more a trade-off than a drawback, and part of the trade-off is the expense of having another software app in addition to DS.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596 Using CFF FAQ: http://www.dsxchange.com/viewtopic.php?t=157872
Rate this response:  
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 679
Location: Malvern, PA
Points: 6425

Post Posted: Wed Dec 27, 2017 1:39 pm Reply with quote    Back to top    

BMC Control-M "about" and usage manuals documentation:
https://docs.bmc.com/docs/display/public/workloadautomation/Control-M+Workload+Automation+Documentation

I checked the links, you don't have to log in to view them.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596 Using CFF FAQ: http://www.dsxchange.com/viewtopic.php?t=157872
Rate this response:  
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42576
Location: Denver, CO
Points: 219102

Post Posted: Wed Dec 27, 2017 2:05 pm Reply with quote    Back to top    

We moved from a POS internal "scheduler" (I use the term loosely) to a full blown enterprise-wide Control-M environment and it does way more things than we would ever need. All for one high high price. Wink

And thanks for the SSI details, familiar with the concept if not the name...

_________________
-craig

I know I don't say this enough, but I like when you talk to me. It's much better than when nobody talks to me. Or when people that I don't like will not stop talking to me.
Rate this response:  
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 679
Location: Malvern, PA
Points: 6425

Post Posted: Wed Dec 27, 2017 2:14 pm Reply with quote    Back to top    

You're welcome, Craig. We have thousands of jobs on multiple platforms -- z/OS, Unix and Windows -- which run nightly and are dependent on variations due to holidays and time constraints. Our ROI for Control-M is rather good. I don't know if BMC has "light" versions for smaller shops, but it's worth a look.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596 Using CFF FAQ: http://www.dsxchange.com/viewtopic.php?t=157872
Rate this response:  
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 106

Points: 1791

Post Posted: Wed Dec 27, 2017 3:51 pm Reply with quote    Back to top    

Thanks gentlemen for your replies, MUCH appreciated. As it is currently we are rather locked down on the software we have for now, I will absolutely keep this in mind for future recommendations to the budgeting committee. For now it sounds like there's not another way to accomplish what I need other than what I already have. In regards to the original question I will absolutely just create a user var that has the project name in each project for now, not a huge deal.

_________________
-Me
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours