Balanced Optimizer - Transform Stage
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 209
- Joined: Fri Jan 09, 2004 1:14 pm
- Location: Toronto, Canada
- Contact:
Balanced Optimizer - Transform Stage
Folks,
One of the DS administrators at my current client had suggested that the GCC compiler is needed before any DS job with a transform stage could be pushed down using the balanced optimizer.
Has anyone had experience with pushing down jobs that included a transformer? Was the GCC compiler needed?
Thanks,
Rob
One of the DS administrators at my current client had suggested that the GCC compiler is needed before any DS job with a transform stage could be pushed down using the balanced optimizer.
Has anyone had experience with pushing down jobs that included a transformer? Was the GCC compiler needed?
Thanks,
Rob
Rob Wierdsma
Toronto, Canada
bartonbishop.com
Toronto, Canada
bartonbishop.com
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The compiler is needed to compile any parallel job with a Transformer in it. This has nothing at all to do with Balanced Optimization.
I don't know whether pushing any of the job's logic into a database can be done without the compiler (my installations have always had it). It would surely depend on whether you're trying to push some of the Transformer stage logic into a database.
I don't know whether pushing any of the job's logic into a database can be done without the compiler (my installations have always had it). It would surely depend on whether you're trying to push some of the Transformer stage logic into a database.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 209
- Joined: Fri Jan 09, 2004 1:14 pm
- Location: Toronto, Canada
- Contact:
Hi lstsaur,
When I say 'Pushing Down Jobs', I mean using the balanced optimizer to push the logic down to the database. The implications are that the data doesn't leave the database, all processing occurs without data crossing the network. The intent is that our jobs will take less resources and perform significantly faster.
IBM does a wonderful job illustrating this feature in the article at the following link: http://www.ibm.com/developerworks/data/ ... connector/
I've had considerable success using similar features in BODS and Informatica. Now that I'm back in the DataStage world, I'm very interested in making this technology work in my latest projects. At this point, the client is still exploring this feature.
Regards,
Rob
When I say 'Pushing Down Jobs', I mean using the balanced optimizer to push the logic down to the database. The implications are that the data doesn't leave the database, all processing occurs without data crossing the network. The intent is that our jobs will take less resources and perform significantly faster.
IBM does a wonderful job illustrating this feature in the article at the following link: http://www.ibm.com/developerworks/data/ ... connector/
I've had considerable success using similar features in BODS and Informatica. Now that I'm back in the DataStage world, I'm very interested in making this technology work in my latest projects. At this point, the client is still exploring this feature.
Regards,
Rob
Rob Wierdsma
Toronto, Canada
bartonbishop.com
Toronto, Canada
bartonbishop.com
-
- Premium Member
- Posts: 209
- Joined: Fri Jan 09, 2004 1:14 pm
- Location: Toronto, Canada
- Contact:
Hi Ray,
While we may find some jobs that we can push down without any logic that would require a transform stage, I don't want to limit myself. My expectation is that I would very quickly get to jobs that have that sort of complexity.
Of course, this will lead to a whole different exercise. We will need to be careful about what functions, data transformations, etc. we use in the transform stage. Each will need to be chosen based on whether there is a database equivalent for us to push down to. Added to that will be the complexity of learning the peculiarities of the tool, i.e. what do we need to do to convince DataStage to push down when it doesn't want to.
Rob
While we may find some jobs that we can push down without any logic that would require a transform stage, I don't want to limit myself. My expectation is that I would very quickly get to jobs that have that sort of complexity.
Of course, this will lead to a whole different exercise. We will need to be careful about what functions, data transformations, etc. we use in the transform stage. Each will need to be chosen based on whether there is a database equivalent for us to push down to. Added to that will be the complexity of learning the peculiarities of the tool, i.e. what do we need to do to convince DataStage to push down when it doesn't want to.
Rob
Rob Wierdsma
Toronto, Canada
bartonbishop.com
Toronto, Canada
bartonbishop.com
-
- Participant
- Posts: 437
- Joined: Fri Oct 21, 2005 10:00 pm
You can't force datastage to push down when it doesn't want to do so. Your only option would be to manually manipulate the job if it didn't push as much logic into the database as you were hoping.
Keith Williams
keith@peacefieldinc.com
keith@peacefieldinc.com
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Within the Balanced Optimizer dialog you can specify the range of stages to push into the source/target database. Have you investigated this feature which, I believe, is new in version 9.1?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 209
- Joined: Fri Jan 09, 2004 1:14 pm
- Location: Toronto, Canada
- Contact:
Hi Keith,kwwilliams wrote:You can't force datastage to push down when it doesn't want to do so. Your only option would be to manually manipulate the job if it didn't push as much logic into the database as you were hoping.
That's exactly what I'm talking about. Determining the tricks needed to manually manipulate the job is going to be where all the fun is.
Rob
Rob Wierdsma
Toronto, Canada
bartonbishop.com
Toronto, Canada
bartonbishop.com
-
- Premium Member
- Posts: 209
- Joined: Fri Jan 09, 2004 1:14 pm
- Location: Toronto, Canada
- Contact:
Hi Ray,ray.wurlod wrote:Within the Balanced Optimizer dialog you can specify the range of stages to push into the source/target database. Have you investigated this feature which, I believe, is new in version 9.1?
We're still getting access to 9.1. The admin hasn't yet gotten the GCC compiler installed, this is the main holdup at this point. The initial debate with the admin was whether the compiler was needed to do any push down at all.
At this point it's still all theoretical for me.
Rob
Rob Wierdsma
Toronto, Canada
bartonbishop.com
Toronto, Canada
bartonbishop.com