Warnings after changing the partitioning

[richdhan]

Hi,

We have a parallel job for dealing with SCD. Here we have
CDC stage(to find new,updated or deleted records),
Join stage(to find existing records which are updated/deleted),
Lookup stages(to get effective_in_dts for new records),
Funnel stage(to combine both new and updated records)

Initially our job used to run on multinode. We thought we could improve the performance by setting data partitioning. So we set the partitioning type to HASH for input links to CDC stage and Join stage and we set the partitioning to ROUND ROBIN for input links to Lookup stage and Funnel stage.

When we run the jobs we get the desired output but along with that we get additional warnings which we never got before partitioning.

The following 2 warnings comes from 2 stages where we are changing the partition to round robin

Code: Select all

LKPS_ODS_GSDB: When checking operator: Operator of type "APT_LUTProcessImpl": Will partition despite preserve-partitioning flag on dataset on input port 0
FUNL_InsertsUpdates: When checking operator: Operator of type "APT_FunnelOperator": Will partition despite preserve-partitioning flag on dataset on input port 0

The following 2 warnings are repeated 5 times and we could not figure out from which stage it is generated.

Code: Select all

APT_ParallelSortMergeOperator(0),0: WARNING: ParallelSortMerge is combined with its input.
APT_ParallelSortMergeOperator(0),0: WARNING: Partitioning for combined operators is straight-through, so ParallelSortMerge will do nothing.

Eventhough we get the desired results we dont want to see these warnings. If these can be avoided it would be great.

If someone can throw some light on this it would be of great help to us.

Thanks in advance
Rich

[leo_t_nice]

The first warnings, about the partitioning, can be removed by changing the Preserve Partitioning flag on the Stage-Advanced tab. If you set this to Clear then the message should disappear (this flag indicates to following stages that it should try to maintain the partitioning of the previous stage).

The second warnings can be removed in the same Stage-Advanced tab, by changing the Combinability mode. Change this to "Dont combine". You may need to do this on both the join and funnel stages.

I hope this helps.

[richdhan]

Hi Leo,

Thanks for the post.

We set Preserve Partitioning flag to Clear before the Lookup stage and Funnel stage and the following warnings got eliminated.

Code: Select all

LKPS_ODS_GSDB: When checking operator: Operator of type "APT_LUTProcessImpl": Will partition despite preserve-partitioning flag on dataset on input port 0 
FUNL_InsertsUpdates: When checking operator: Operator of type "APT_FunnelOperator": Will partition despite preserve-partitioning flag on dataset on input port 0 

The second warnings can be removed in the same Stage-Advanced tab, by changing the Combinability mode. Change this to "Dont combine". You may need to do this on both the join and funnel stages.

The second warning was eliminated by changing the combinablity mode to 'Dont combine" not for join and funnel stage as you had mentioned but for CDC stage and Transformer stage. We did this through trial and error method beacuse we did not know which stage APT_ParallelSortMergeOperator(0) was referring to.

I have some questions.

1. How to know which stage APT_ParallelSortMergeOperator(0) is referring to?
2. I went through the documention on Preserve Partioning and Combinability mode on Parallel Job developers Guide but do you know what happens internally in Datastage when you set different options and how it affects the performance.

Thanks in advance
Rich

[bobyon]

I too woul like to have answers to these questions. It's been a while since they were originally posted so I hope no one minds me reposting them

I have some questions.

1. How to know which stage APT_ParallelSortMergeOperator(0) is referring to?
2. I went through the documention on Preserve Partioning and Combinability mode on Parallel Job developers Guide but do you know what happens internally in Datastage when you set different options and how it affects the performance.

[/quote]