How to prevent datastage jobs from retrieving source info?

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
jason2018
Participant
Posts: 5
Joined: Tue Apr 03, 2018 8:39 pm

How to prevent datastage jobs from retrieving source info?

Post by jason2018 »

I found that datastage jobs would detect the source table, retrieving information through the connector like db2 and informix connector, not just execute by the properties of fields I defined in the "columns tab".
I found that while using funel and change capture stage.
Funel stage requires all sources have the same column definition, but even I defined the same columns in datastage for all connectors, it still throws an error that the actual tables have different column definitions.
And so is the change capture stage. If I use
change mode=Explicit Keys, All Values
it always throws an error
"A field cannot be both a key and a value"
But if I use
change mode=Explicit Keys & Values
, it runs ok. And I guess that's because the stage can't tell the value fields or retrieves the wrong information from the connector.
And I still found that: only the informix connector causes the problem for change capture stage. The ODBC connector, oracle connector doesn't (I guess it can't retrieve information from the API or due to some other thoughts), and DB2 connector doesn't because it would retrieve the right info.
And, all I want to know is that How to prevent datastage jobs from retrieving source info, no matter the connector is informix,oracle or db2 or any other type.
Thanks all
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

The DB2 Connector stage has a few properties related to Schema Reconciliation that you can toggle but I doubt those are going to help you much in this case.

A couple of ideas:

1) update your job properties and stage properties to disable Runtime Column Propagation, in case it is enabled, just to make sure it is not introducing any unexpected surprises

2) insert a Transformer stage in between your database stage and change capture stage where you do any necessary explicit type conversions and the like, because it sounds like you are really wanting to define the columns in your job a bit differently than how they are defined in the database
Choose a job you love, and you will never have to work a day in your life. - Confucius
jason2018
Participant
Posts: 5
Joined: Tue Apr 03, 2018 8:39 pm

Post by jason2018 »

thanks for replying. I tried that before, both, but didn't work. You get my point, I want to define the "keys" on my own, even if there is no unique constraints on the source table
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

If you have Runtime Column Propagation enabled on the Connector stage (Outputs tab, Columns), disable it.

A sometimes unwanted side effect of using RCP in a source stage that retrieves data from a database table is effectively to generate a "SELECT * FROM..." query.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply