DSXchange

Hi,

I have multiple Datastage jobs that need to be created for loading into the same target table, basically, due to parallel development requirement to save time.
So, I would have around 40 column derivations done by developer1 in one Datastage job and another developer would create second Datastage job for around 30 columns. It would require some manual integration at the end of development to combine the 2 jobs - by adding the columns and transformations into a single job.
Please let me know if I can combine both the job dsx instead of opening the Datastage job and editing it?

Thanks

Okay. First off welcome aboard but secondly welcome to your very own post. I'm guessing in your dsx searching you found the "cutter" discussion but you need to be out here on your own rather than at the tail end of that conversation. Also moving it here, although it seems like a general type question, I suspect it will spin off into more Parallel specific areas.

I'm also going to start the voting off with a "no". Or perhaps a highly qualified "yes" with a warning about the work it would take to combine the two jobs via dsx surgery. We'll see what others think.

And to be honest, the idea of splitting a job up between two developers each doing half of the columns sounds like a bad idea, or at least not an ideal solution to me. Seems to me, in an effort to "save time" you are creating more work for your team and would actually take longer doing it that way. But that's just my opinion.

So you've ruled out RCP for this because all of your columns have required transformations that need to be performed on them?

dsx hacking is really only useful for very simple things like copying job parameters from one job to another ( a feature that is sadly lacking), or the occasional overarching search and replace of 1 word or something.

I don't understand enough of your design, but it seems to me you could use a shared container for whatever is shared across the 'multiple jobs' and you could use a join stage (perhaps, this may need to be in another reusable container) and a transformer to put the 30 flow and 40 flow back together and crank out the final product. It may be a little clunky, but live and learn -- if it works and isnt too slow or hard to manage you can salvage this one and think up a better approach next time.

Thanks for the replies.
We did rule out RCP since the column derivations and lookup tables/columns for the columns are different.
Also we are using BDFS stage and want to reduce the number of read/write across multiple jobs. Hence, we want to develop a single job with all the column calculation in the same job (and not use 3 jobs at the end of development - not to use 2 jobs to develop and 1 to combine).

So... do you have a resolution here?

chulett wrote:So... do you have a resolution here?

We are still combining the jobs manually, any further suggestions would help

I like to Play around with editing and modifying things in exported .dsx or .XML files when necessary. It is finicky work and it doesn't take much to corrupt Jobs completely and irrevocably by changing the files.

The internal structure of an Export file doesn't lend itself to merging, the effort in writing some code to do this type of work is going to be far more than getting someone to manually cut-and-paste in the DataStage designer.

if you do anything manually more than 2 or 3 times, its time to automate it. If none of the datastage ideas works for you, write something that will do the surgery for you.

And its not always that much code. The one I did to transfer job params from one job to another took like 2 hours or less from start to finish. Time you have manually hacked on 3 or 4 jobs you could have done that.

You can copy/paste between Transformer stages. They don't even have to be in the same project!