DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
rajalaa
Participant



Joined: 25 Mar 2014
Posts: 4
Location: Chennai
Points: 48

Post Posted: Wed Jun 27, 2018 2:34 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Additional info: DataStage version: 11.5
Hi,

I have multiple Datastage jobs that need to be created for loading into the same target table, basically, due to parallel development requirement to save time.
So, I would have around 40 column derivations done by developer1 in one Datastage job and another developer would create second Datastage job for around 30 columns. It would require some manual integration at the end of development to combine the 2 jobs - by adding the columns and transformations into a single job.
Please let me know if I can combine both the job dsx instead of opening the Datastage job and editing it?

Thanks
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42796
Location: Denver, CO
Points: 220596

Post Posted: Wed Jun 27, 2018 7:15 am Reply with quote    Back to top    

Okay. First off welcome aboard but secondly welcome to your very own post. I'm guessing in your dsx searching you found the "cutter" discussion but you need to be out here on your own rather than at the tail end of that conversation. Also moving it here, although it seems like a general type question, I suspect it will spin off into more Parallel specific areas.

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42796
Location: Denver, CO
Points: 220596

Post Posted: Wed Jun 27, 2018 9:08 am Reply with quote    Back to top    

I'm also going to start the voting off with a "no". Or perhaps a highly qualified "yes" with a warning about the work it would take to combine the two jobs via dsx surgery. We'll see what others think.

And to be honest, the idea of splitting a job up between two developers each doing half of the columns sounds like a bad idea, or at least not an ideal solution to me. Seems to me, in an effort to "save time" you are creating more work for your team and would actually take longer doing it that way. But that's just my opinion.

So you've ruled out RCP for this because all of your columns have required transformations that need to be performed on them?

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
UCDI



Group memberships:
Premium Members

Joined: 21 Mar 2016
Posts: 336

Points: 3396

Post Posted: Wed Jun 27, 2018 10:38 am Reply with quote    Back to top    

dsx hacking is really only useful for very simple things like copying job parameters from one job to another ( a feature that is sadly lacking), or the occasional overarching search and replace of 1 word or something.

I don't understand enough of your design, but it seems to me you could use a shared container for whatever is shared across the 'multiple jobs' and you could use a join stage (perhaps, this may need to be in another reusable container) and a transformer to put the 30 flow and 40 flow back together and crank out the final product. It may be a little clunky, but live and learn -- if it works and isnt too slow or hard to manage you can salvage this one and think up a better approach next time.
Rate this response:  
Not yet rated
rajalaa
Participant



Joined: 25 Mar 2014
Posts: 4
Location: Chennai
Points: 48

Post Posted: Tue Jul 03, 2018 1:39 am Reply with quote    Back to top    

Thanks for the replies.
We did rule out RCP since the column derivations and lookup tables/columns for the columns are different.
Also we are using BDFS stage and want to reduce the number of read/write across multiple jobs. Hence, we want to develop a single job with all the column calculation in the same job (and not use 3 jobs at the end of development - not to use 2 jobs to develop and 1 to combine).

_________________
Thanks
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42796
Location: Denver, CO
Points: 220596

Post Posted: Tue Jul 03, 2018 8:09 am Reply with quote    Back to top    

So... do you have a resolution here?

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
rajalaa
Participant



Joined: 25 Mar 2014
Posts: 4
Location: Chennai
Points: 48

Post Posted: Fri Jul 13, 2018 1:27 am Reply with quote    Back to top    

chulett wrote:
So... do you have a resolution here?


We are still combining the jobs manually, any further suggestions would help

_________________
Thanks
Rate this response:  
Not yet rated
ArndW

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup

Joined: 16 Nov 2004
Posts: 16318
Location: Germany
Points: 92566

Post Posted: Fri Jul 13, 2018 2:06 am Reply with quote    Back to top    

I like to Play around with editing and modifying things in exported .dsx or .XML files when necessary. It is finicky work and it doesn't take much to corrupt Jobs completely and irrevocably by changing the files.

The internal structure of an Export file doesn't lend itself to merging, the effort in writing some code to do this type of work is going to be far more than getting someone to manually cut-and-paste in the DataStage designer.

_________________

Image
Rate this response:  
Not yet rated
UCDI



Group memberships:
Premium Members

Joined: 21 Mar 2016
Posts: 336

Points: 3396

Post Posted: Fri Jul 13, 2018 2:07 pm Reply with quote    Back to top    

if you do anything manually more than 2 or 3 times, its time to automate it. If none of the datastage ideas works for you, write something that will do the surgery for you.

And its not always that much code. The one I did to transfer job params from one job to another took like 2 hours or less from start to finish. Time you have manually hacked on 3 or 4 jobs you could have done that.
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54407
Location: Sydney, Australia
Points: 295092

Post Posted: Wed Jul 18, 2018 7:39 pm Reply with quote    Back to top    

You can copy/paste between Transformer stages. They don't even have to be in the same project!

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours