Tranform runs too long!

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
Marley777
Premium Member
Premium Member
Posts: 316
Joined: Tue Jan 27, 2004 3:26 pm

Tranform runs too long!

Post by Marley777 »

Please Help! :?

Hello, I have a transformer problem and was wondering if anyone has experienced the same thing.

I'm simply mapping my input to my ouput and creating an additional field based on the value of another field. I use an if statement in the derivation column of the new field to check the value of an existing field. The value in the new field is determined by the value in the existing field. The if statement is big, but can't beleive it would slow things down that much. My tranformer never stops running. I'm only processing 50,000 records. My co workers are experiencing the same things with their Basic Transformer stages. All of our transformer stages are different in what they do, but they all run forever. We are not sure what the problem is and need help.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

There's definitely a problem here. You should be able to do 50000 rows in about a minute or two (depending on hardware) if source and target are text files. Maybe five.
Perhaps the source file lacks an end-of-file character, and your BASIC Transformer stage is waiting forever for it. (This is how this stage type is signalled that everything is complete; it issues a "get next row" call and has "no data found" returned.)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
bigpoppa
Participant
Posts: 190
Joined: Fri Feb 28, 2003 11:39 am

Tranform runs too long!

Post by bigpoppa »

I agree with Ray. If you are not able to solve the problem, could you post the generated basic code?

Thanks
BP
gh_amitava
Participant
Posts: 75
Joined: Tue May 13, 2003 4:14 am
Location: California
Contact:

Post by gh_amitava »

Hi,

This problem may appear anytime if you are using Basic Transformer in a parallel job in datastage 7.0. It is suggested by DataStage that instead of using Basic transformer, use PX Transformer. This is due to the fact that Basic transformer is not a part of orchestrate engine and for this you may face the mutex error or the job may wait for long.

Regards
Amitava
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Can you give us a reference for this statement?
Is it on the Ascential web site?
Is there an e-case for it?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
ariear
Participant
Posts: 237
Joined: Thu Dec 26, 2002 2:19 pm

Post by ariear »

There are many problems using a Basic transformer in PX and that's only one of them. There's seems to a problem when using a Basic transformer after a parallel Sort also. The good news, Ray ,are that you can do 50000 rows a second ! :D
Post Reply