Quality Stage Reference match

[vijaydasari]

We have customer information in two databases , now I am doing standardization and un duplication using Quality stage
And loading into customer master table (this table holds information of customer key , first name , last name etc..
address details we are maintaining in a separate table)

During initial load process I am going to process all records and make them master , dups and residual data.

But for the incremental load I am going to apply CDC logic and process only new/ changed rows .
as per our business process for changed rows should use the existing customer key.

In this case do I need to use reference match stage for matching purpose ?

Please let me know if you have any additional questions.

[ray.wurlod]

It depends on how you're doing your incremental load. If all you're interested in is matching in the new data then no, use an unduplicate match. If you want to compare the new data against the existing data in the target then use a reference match.

[vijaydasari]

Thank you very much for the response.

Our requirement is to compare new data against existing data.

As per your suggestion, I will use the reference match stage.

I have one question in choosing reference source input to Reference match stage.

In our process I am loading match data and residual data to customer master table.

But this table does not have address details attributes (it has only cust key , name etc..).

I am planning to use standardized dataset created during initial load as reference input dataset and standardized dataset create during incremental load as primary input data source.

Please let me know is this approach works?

[ray.wurlod]

Have you thought about the second and subsequent incremental loads? You will need to refresh your reference data set so that it's effectively maintained as a copy of the target table (as closely as may be supported by the columns you are storing there). But, yes, the design should work subject to that caveat.

[vinodn]

Maintaining reference dataset is a very good point . Thank you very much Ray.