RCP and adding new columns to flow

[juancho]

Hello!

I am trying out Runtime Column Propagation (RCP) using a simple sequential file (reading the metadata from a Schema file), a Transformer and another sequential file to write the data to.

What do i need to do:
Add a date column to the flow.

What do i have:
Just three columns, lets suppose i have this file:

col1,col2,col3
100,200,300

i read it using RCP and a schema file then process it and i want this as output:

date,col1,col2,col3
20090303,100,200,300

Is there any way to do that using RCP? because i have like 40 jobs where the only thing i have to do is this (well that and write from text to dataset) and i would love to save the development time.

Currently, adding a single column in the transformer stage writing the date only returns:
20090303

and none of my original schema read columns.

Any help? maybe another way to do it?

Thanks!

[Sainath.Srinivasan]

Add a column generator instead of transformer.

That can use schema.

[miwinter]

Have you verified that RCP is enabled at the project level?

[juancho]

I have verified that RCP is enabled.

I am trying the column generator option, but how do i do so it uses certain function while generating the data?

the only option i see is
Cycle (cycles thru several options)
Alphabet (chooses one letter each time from a chosen string)


Thanks in advance!

[miwinter]

What is the 'certain function' you want to apply? From your example, I think we (at least I was) assumed you were using a literal each time. If it's more complex than that, this stage won't suit as it's intended to produced test data really, providing the same value over and over or working around a list over and over. It can't employ transform functions if you are wanting to do something fancier with your date you want to add.

[juancho]

Well i need to add the current date to the new column, so that is as complicated as it gets...

any ideas? i am fiddling with Funnel to see if i can get anything done by merging two flows...

[Sainath.Srinivasan]

Modify stage !!??

[juancho]

But for Modify Stage i need to define the columns i KEEP and DROP, i cannot add a new column there and i want to use RCP.

Maybe i'm wrong about it...

[juancho]

I think the only solution is to process the file as a whole huge record and then add another column (having the current date) at the transformer level, this should be done in a first job, the second job should use RCP and i will already have [date]<separator>[original data], hence i will be able to use only one job.

And since the first job only reads the data as one huge record instead of delimited, i dont need to define any special metadata for any of the files.

that way with three jobs (the third being a dataset to oracle inserter) i am able to process all the files, only using schema definitions.

If anyone thinks of a better solution i am still open to your help!

[Sainath.Srinivasan]

Insert one extra dummy column in the column generator stage and drop it in the modify.

[juancho]

Insert one extra dummy column in the column generator stage and drop it in the modify.

But i wont be able to fill that column with the current date, am i right?

[ray.wurlod]

You are not right.

The Generator property folder for a date data type includes Use current date as an available property to add.

[juancho]

Thanks!

Solved it.

[aymancherif]

hello juancho,

if u please can tell me how u solved this issue because i have the same situation here, i have huge number of jobs so i did enable the RCP and i want to add some new columns during transformation.

thanks

[ray.wurlod]

Welcome aboard. How do you expect juancho to know what U (one of our other posters) did?

Please strive for a professional standard of written English on DSXchange. It is a web site, not a mobile telephone, so there is no need for SMS-style abbreviations. Many of our posters (and lurkers) do not have English as their first language, so even good English is enough of a challenge for them.