Complex Data Rules in IA

This forum contains ProfileStage posts and now focuses at newer versions Infosphere Information Analyzer.

Moderators: chulett, rschirm

Post Reply
cosmok
Participant
Posts: 6
Joined: Wed Apr 27, 2011 8:55 pm

Complex Data Rules in IA

Post by cosmok »

‪I can't figure out how to implement complex logic in data rules. The logic builder and free form editor don't seem to support things like distinct, group by, or sub queries.‬

‪For example say I had the following SQL query:‬

Code: Select all

‬
‪SELECT c1.Name‬
‪FROM Customers c1‬
‪WHERE c1.CustomerID NOT IN (SELECT DISTINCT c2.CustomerID ‬
‪					FROM Customers c2‬
‪					JOIN Orders o ON c2.CustomerID = o.CustomerID‬
‪					WHERE c2.Gender = 'Male'‬
‪					AND o.ProductCode = 'ABC')‬
‪


‪I tried using the 'not in_reference_column' operator in IA data rules, and set the reference table up as a virtual table in IA (to perform the subquery). However, you can't seem to create a virtual table that joins multiple tables together. So in the case of having a more complex subquery (which I do) virtual tables don't look like an option.‬

‪I guess I could implement a view, but it may not be possible to alter the source database, and its not desirable - it would be better to have all the data quality logic inside IA.‬

‪Another option I guess is to perform a left/right anti semi join. Eg.‬

Code: Select all

‬
‪SELECT c1.Name‬
‪FROM Customers c2‬
‪JOIN Orders o ON c2.CustomerID = o.CustomerID AND c2.Gender = 'Male' and o.ProductCode = 'ABC'‬
‪RIGHT OUTER JOIN Customers c1 ON c1.CustomerID = c2.CustomerID ‬
‪WHERE c2.Name IS NULL‬
‪


‪But since I would be joining a table to itself I'd need to be able to define a table alias - again something IA doesn't seem to support.‬ I don't think the join condition can be anything more than key1 = key2 either.

Am I ‪having a mental blank here, or is it not possible to implement this kind of complexity in IA data rules?‬
Brett
Participant
Posts: 12
Joined: Tue Sep 08, 2009 4:57 pm
Location: Sydney

Post by Brett »

I have had similar problems and have currently resorted to creating table views designed for testing some of the more complex business rules. My environment is discrete from the main system and I have the authority to do this.

You can create simple join conditions in the rules editor, there is also the option of virtual tables to add filters to tables. I did submit a request to IBM to allow for a better query design function to drive the data selection for the rules builder.

We have some fairly complex queries that define the current version of say the customer contact record and for the moment I am staying with the views as the best solution. Which I then use to run business rules against to measure completeness of data etc.

Not sure what more experienced users have done - would be interested in a better solution ?

Brett.
wendyxu_2009
Participant
Posts: 4
Joined: Wed Aug 05, 2009 10:10 pm

Post by wendyxu_2009 »

Brett wrote:I have had similar problems and have currently resorted to creating table views designed for testing some of the more complex business rules. My environment is discrete from the main system and I have the authority to do this.
Brett, would you please explain a bit more in detail how to use table view to return 'distinct', 'group by' functions?

For example, for base table A and pair table B, after foreign key and referential integrity analysis, how to select distinct id (e.g. customer id) for all orphan records? Thanks!
Brett
Participant
Posts: 12
Joined: Tue Sep 08, 2009 4:57 pm
Location: Sydney

Post by Brett »

To clarify - These views are not built inside IA -

As I have an offline environment and full control of the database - I am able to build and construct subsets of data, to drive areas of the rule base I create - such as current address quality for an active client. This way I can build the views, run the rule packs and clean up through scheduled activities. Sorry if I gave the impression I had done this in IA - I did try filtering data as virtual tables where you can apply where clauses and then doing any joins on these virtual tables in the rule editor but this became very hard to manage.
cosmok
Participant
Posts: 6
Joined: Wed Apr 27, 2011 8:55 pm

Post by cosmok »

Thanks Brett. I suspect I will have to go down the View path. Or another possibility might be to build rules up using Virtual Tables, and multiple Data Rules/Rule Sets.
bkamila
Premium Member
Premium Member
Posts: 29
Joined: Tue Jun 14, 2005 5:01 pm
Location: Mascot

Post by bkamila »

Hi All in this Post,
I am struggling with the same issues that you all are facing.

Brett, you have full control over databse for which you are able to develop data subset. Those who don't have control over databse, the solution is virtual table. It leads to my next question -

Does it require to create separate virtual table for each sub-set of data? or create one virtual table then create filter equivlent to group by or distinct stated in this post?

It alludes to another question, which information analyzer role has virtual table creation previlege out os exisiting roles in 8.5?
BANA KAMILA
BI, Data Warehouse and Database Consultant
Post Reply