We have a parallel job for dealing with SCD. Here we have
CDC stage(to find new,updated or deleted records),
Join stage(to find existing records which are updated/deleted),
Lookup stages(to get effective_in_dts for new records),
Funnel stage(to combine both new and updated records)
Initially our job used to run on multinode. We thought we could improve the performance by setting data partitioning. So we set the partitioning type to HASH for input links to CDC stage and Join stage and we set the partitioning to ROUND ROBIN for input links to Lookup stage and Funnel stage.
When we run the jobs we get the desired output but along with that we get additional warnings which we never got before partitioning.
The following 2 warnings comes from 2 stages where we are changing the partition to round robin
Code: Select all
LKPS_ODS_GSDB: When checking operator: Operator of type "APT_LUTProcessImpl": Will partition despite preserve-partitioning flag on dataset on input port 0
FUNL_InsertsUpdates: When checking operator: Operator of type "APT_FunnelOperator": Will partition despite preserve-partitioning flag on dataset on input port 0
The following 2 warnings are repeated 5 times and we could not figure out from which stage it is generated.
Code: Select all
APT_ParallelSortMergeOperator(0),0: WARNING: ParallelSortMerge is combined with its input.
APT_ParallelSortMergeOperator(0),0: WARNING: Partitioning for combined operators is straight-through, so ParallelSortMerge will do nothing.
If someone can throw some light on this it would be of great help to us.
Thanks in advance
Rich