manage lookup table by datastage

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
and
Participant
Posts: 14
Joined: Thu Oct 05, 2017 5:36 am

manage lookup table by datastage

Post by and »

hi

i'm try to find info about using datastage or its components for managing (add, delete, update) lookup table.

are there some tools (may be with GUI ) for this ?

or may be preferable way to administer such things are database, files ...

any ideas....


thanks
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Welcome aboard.

You'll need to be a little more specific.

DataStage is certainly able to maintain tables. However, to use a data source as a lookup table with the Lookup stage, the lookup table is read into memory when the job starts, and cannot be maintained in memory.

The Slowly Changing Dimension stage may assist your quest, however.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
and
Participant
Posts: 14
Joined: Thu Oct 05, 2017 5:36 am

Post by and »

ray.wurlod wrote:the lookup table is read into memory when the job starts
is there some tool (may be with GUI ) for add/delete some data in it before i will use it ?

in general i want to have some lookup ( may be mapping table ) managed by datastage. of course i can have some table in database then read it by some job and write to some file then use this file in lookup
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

You are on point up until the "write to some file" part. The lookup would be read into memory directly from the database table.
-craig

"You can never have too many knives" -- Logan Nine Fingers
UCDI
Premium Member
Premium Member
Posts: 383
Joined: Mon Mar 21, 2016 2:00 pm

Post by UCDI »

or directly from a dataset / file. But you don't need both, either a file or a table.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Let me make it really clear. As a general rule DataStage parallel jobs do not allow the one object to be written to and read from in the same job. This is what's called a blocking operation.

The Slowly Changing Dimension stage is the exception that proves the rule.

You have the alternative of using server jobs, which DO allow blocking operations (e.g. to hashed files) provided that you disable the memory caches.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
and
Participant
Posts: 14
Joined: Thu Oct 05, 2017 5:36 am

Post by and »

thanks for note

ok
i try to describe in other view

for example

i have parallel job ( job1 ) which use some lookup ( L1 ).
also i have another parallel job ( job2 ) which fill file for L1 of job1 from some source ( S1 )


so i want to know some ways and tools for working with S1

as idea
1) S1 can be some database table and in job2 i get data from this table and put it in file for L1
2) S1 can be some text file ( csv ... ) and also in job2 i get data from this file and put it in file for L1

so for 1 case i can create some GUI and dynamically add/delete rows in database table
as well for case 2

so is where some tools / ways for working with data ( database table or text file) from datastage ?


thanks
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

The only tool you have in DataStage is the DataStage job.

One job can populate L1. You would run this job whenever S1 has been changed.

The other job uses L1 as the source for a reference link to a Lookup stage. Once this job has started, it takes a snapshot of L1 into memory; there is no way that this memory image can be manipulated while this job is running.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
and
Participant
Posts: 14
Joined: Thu Oct 05, 2017 5:36 am

Post by and »

ray.wurlod wrote:The only tool you have in DataStage is the DataStage job.
it's a pity


thanks
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

:?
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply