DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
vbr_03
Participant



Joined: 18 Apr 2012
Posts: 2

Points: 24

Post Posted: Wed Aug 08, 2018 10:27 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hi ,

Is there any way to restart a parallel job to load the data from last failure point?

_________________
VIJ
leandrohmvieira
Participant



Joined: 02 Sep 2015
Posts: 44
Location: Brasilia, Brazil
Points: 438

Post Posted: Wed Aug 08, 2018 1:39 pm Reply with quote    Back to top    

Sequence jobs does have some checkpoint functionality, which allow a sequence to restart from it.

Parallel Jobs and Server Jobs does not have any features like this. Can you provide some details of your problem?

_________________
Leandro Vieira

Data Expert - Brasilia, Brazil
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54407
Location: Sydney, Australia
Points: 295092

Post Posted: Thu Aug 09, 2018 12:22 am Reply with quote    Back to top    

Short answer, no.

You may be able to design jobs with a certain degree of restartability but, in general, the amount of effort required would make it not worthwhile.

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42792
Location: Denver, CO
Points: 220559

Post Posted: Thu Aug 09, 2018 6:37 am Reply with quote    Back to top    

Right, restartable jobs are certainly possible, I've always striven for atomic level job designs ('single units of work') to allow them to be restartable with little or no human intervention. I've posted high level notes here in the past describing the 'framework' we're using now to support that.

Restarting from the point of failure? That's a whole 'nuther kettle of fish, especially if there's any kind of complexity in the job design and would generally require some kid of... let's say "compromises"... with regard to job speed.

(technically, the tool I'm using now has a magical checkbox to enable that functionality but I've yet to try/playwith/trust any such feature)

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
Joel in KC
Participant



Joined: 10 Aug 2018
Posts: 3

Points: 30

Post Posted: Tue Aug 14, 2018 3:12 pm Reply with quote    Back to top    

Please let me know where I can find your framework and and "single unit of work" as we are trying to move to this type of usage, rather than the huge, complex systems that need re-starting,,,appreciate your time. New to the board. Thx again
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42792
Location: Denver, CO
Points: 220559

Post Posted: Tue Aug 14, 2018 7:24 pm Reply with quote    Back to top    

Both are mentioned here with some high level details for the framework. Hope it helps. As noted there, would really be interested to see if anyone has done anything like that in DataStage, mine is an Informatica implementation which makes it a tad easier.

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54407
Location: Sydney, Australia
Points: 295092

Post Posted: Tue Aug 14, 2018 8:22 pm Reply with quote    Back to top    

Where I need this functionality I, like Craig, create small atomic units of work as DataStage job, and make use of the restartability capability of sequence jobs to handle that. No point in re-invent ...

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 707
Location: Malvern, PA
Points: 6719

Post Posted: Wed Aug 15, 2018 8:52 am Reply with quote    Back to top    

High-level error handling design is where restartability is identified. Error handling is a part of the definition of the unit of work.

Example:

1. Download file. If that fails, fix problem and rerun.
2. Process file. If there are no intermediate points of failure -- like commits -- if the process fails fix and rerun.
3. Etc.

DataStage permits jobs that do both functions in one parallel job. If your design does that, you're next step is to rewrite the job to create the separate units of work.

Job Sequence design covers the how and where.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596 Using CFF FAQ: http://www.dsxchange.com/viewtopic.php?t=157872
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours