Transfering sources to a single file

This forum is in support of all issues about Data Quality regarding DataStage and other strategies.

Moderators: chulett, rschirm

Post Reply
mohdtausifsh
Participant
Posts: 14
Joined: Tue Dec 20, 2005 2:29 am

Transfering sources to a single file

Post by mohdtausifsh »

Hi!

I have 11 sources each source has different conditions .I have to retrive 8 fields from each of the sources an the target is flat file having 16 standard fields .I need to map these 8 values coming from sourec to the respective target columns.Can you ppl give an suggestion to arrive to an simple solution,

Thanks
Tauseef
kumar_s
Charter Member
Charter Member
Posts: 5245
Joined: Thu Jun 16, 2005 11:00 pm

Post by kumar_s »

Hi,
welcome Aboard :D

You have asked, in simple, how to do a datawarehouse project :wink:
Do you have you High level design document/Low level Design document. Or only the feild mapping?
Is it all direct mapping?
What type of condition are they?
Do you have any aggregation?
More you provide the detail, more you get the information and answers.

-Kumar
nishant_prakash
Participant
Posts: 27
Joined: Wed Aug 17, 2005 5:18 am

Re: Transfering sources to a single file

Post by nishant_prakash »

Hi Buddy,

The way you have put up the problem, it seems to be quite simple.
Connect all the source links to a transformer stage and if the sources have a common key column then join the records based on that key.
Then based on the source conditions apply the constraints on the source if it is a database stage like oracle.

After the records are joined map them to respective target columns in the transformer stage and write to the output flat file
mohdtausifsh
Participant
Posts: 14
Joined: Tue Dec 20, 2005 2:29 am

Post by mohdtausifsh »

But my client wants me to use Single source table .And he wants me to check the data coming from the table in a file which has all the possible combinations of the conditions aas a look up.BUt the proble is that we can stisfy the conditions but we cannot direct which source value should go to which target column.As satisfying the conditions would'nt be a problem.But directing them to the respective columns will definitely be a problem.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

No it won't. In a Transformer stage you can direct whatever columns you like to whatever output links you like.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
mohdtausifsh
Participant
Posts: 14
Joined: Tue Dec 20, 2005 2:29 am

Post by mohdtausifsh »

I have 11 source and out of these you have 8 colums from each source coming out to the target based on the condition.So i can take all the possible colums from the source .filter that columns based on the conditions in the flat file.Now the problem is that for one condition there are different combinations of records going in to the target .According to the scenario in the Transfrormer stage there are !2 possible combinations in the input link and 16 columns in the target.So 8 columns should go to the target based on the condition.We can direct the values if there or one or two coinditions.But there are some situations where the conditions are same but data going to the target is different .What should i do
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Document all the possibilities, and the columns that need to be transferred in each case.

The result should be a comprehensive target-from-source mapping document.

The process of creating this document will clear your mind as to how to implement the ETL.

I wasn't clear whether you have eleven sources or a single source table. Is there some mechanism whereby the eleven sources might be combined into a single source - for example cat command? That is, are the eight columns in each of your data sources the same?

What is the format of this sequential file that contains the conditions?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
mohdtausifsh
Participant
Posts: 14
Joined: Tue Dec 20, 2005 2:29 am

Post by mohdtausifsh »

can you suggest some fonctions or routiens to direct the values from the source to the respective columns in the target based on the conditions.Because right now the main problem is that we can satisfy the conditions creating a file having all the possible combinations.But in the transformer how can we direct the values with out mapping them physically.Cos we have to map some how using routines or transforms.
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

Perhaps it might help to give an example of your requirements to illustrate your question. I am assuming that your question is more complicated, as you have self-evaluted yourself as an expert so it is unlikely that you are asking about the base DataStage transform functionality of mapping columns based on conditions.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

mohdtausifsh wrote:can you suggest some fonctions or routiens to direct the values from the source to the respective columns in the target based on the conditions.
No.
Because you have not documented the algorithms required.
mohdtausifsh wrote:Because right now the main problem is that we can satisfy the conditions creating a file having all the possible combinations.But in the transformer how can we direct the values with out mapping them physically.Cos we have to map some how using routines or transforms.
Who said you have to do it this way?
What's wrong with regular constraint expressions?
(These can, of course, incorporate routines and/or transforms.)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply