DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message

Group memberships:
Premium Members

Joined: 22 Apr 2011
Posts: 59

Points: 517

Post Posted: Fri Oct 25, 2013 1:15 pm Reply with quote    Back to top    

DataStage® Release: 8x
Job Type: Parallel
OS: Unix
Additional info: Version 8.7
Hi All,

I have designed a Reference Match job where the Data will contain around 10k - 100k records and my reference set right now contains around 3 million and is expected to grow.

The match is being done on the first and last names and the blocking is on the NYSIIS values of the first and last name. But I am getting the following warning:

The number of records in the reference block with key NYSIIS_LAST_NAME(HAL) exceeds the maximum number specified of 10000.
All records in the block will be treated as residuals

The IBM site: http://www-01.ibm.com/support/docview.wss?uid=swg21409638
Suggests that we dont update the Overflow values (which also can have a max of only 40k) but instead limit the number of records with invalid or default values.
But I am filtering all the invalid values much before it reaches Matching.

Is there any other way to prevent these records from becoming residual or impact the matching process? Would be grateful if anyone can point me in the right direction.


Premium Poster

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54524
Location: Sydney, Australia
Points: 295662

Post Posted: Fri Oct 25, 2013 1:58 pm Reply with quote    Back to top    

You need to reduce the number of records per block. Usually the way to do this is to add one or more additional blocking columns. Think about the implications. If you have 10,000 record in a bl ...

IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours