lookup based on rules

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
jasper
Participant
Posts: 111
Joined: Mon May 06, 2002 1:25 am
Location: Belgium

lookup based on rules

Post by jasper »

For an aggregation of (telephony)call-data I need to find the type of call.
This is based on the first digits of the number being called(usually called the significant digits). Problem however is that the number of digits that matters is not always equal. sometimes first 2 is enough, but it can go up till the 8 first that can make a difference.
The way it is done now is to create all necessary substrings (2char,3char,...8char) and then do the 7 lookups. After these are done logic is applied to find the correct one(if 8char<>NULL, use this, else if 7char<>NULL,....) Since this is done for daily millions of records this takes too much time.

Any ideas on other ways of working?
T42
Participant
Posts: 499
Joined: Thu Nov 11, 2004 6:45 pm

Post by T42 »

Input -> Transformer (make multiple fields - phone[1,2] -> twodigitphone; phone[1,3] -> threedigitphone; et cetera) -> Lookups.

With the lookups, use reject links to push non-matches to other lookups. Start with small digits first (as there's less matching obviously), unless your business rules require the opposite.

Do you only need to make one match to confirm this?
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

I once created a bitmapped approach - if the bit in the ordinal position represented by the numeric value of the code exists then it was a valid area code. Very quick. Did it in a server job, but it should be quite possible in PX.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
jasper
Participant
Posts: 111
Joined: Mon May 06, 2002 1:25 am
Location: Belgium

Post by jasper »

T42,
thanx for the suggestion, this is however something I've allready tried. Since business rules require the opposite approach (first 8, then 7,...) This actually takes longer then doing all lookups in one lookup-stage and apply the logic later.


Ray,
I'll try your suggestion, thanx.
Post Reply