How can perform incremental loading without using key

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
skdubey.bi
Participant
Posts: 28
Joined: Mon Nov 28, 2005 1:23 am

How can perform incremental loading without using key

Post by skdubey.bi »

I have a flat files source and it does'nt have any key to comapre record(Like business key).How i can perform incremental loading to the corresponding target.(targate is also a flat file).please post needfull.


thanks in advance,
satish dubey
loveojha2
Participant
Posts: 362
Joined: Thu May 26, 2005 12:59 am

Post by loveojha2 »

Why do you want to go for an incremental load :?:

For the case all the columns would need to be marked as key (or the concatenated string of the whole record) (but I won't advise you for that.)
Success consists of getting up just one more time than you fall.
djm
Participant
Posts: 68
Joined: Wed Mar 02, 2005 3:42 am
Location: N.Z.

Post by djm »

Satish,

The solution I am suggesting is on the basis that the following understanding is correct. Namely, that you have two files:
A: The "master" which represents the current state and
B: The file holding records that may have potentially changed

You want to take only those records in B that are not in A and append them to A.

This solution is based on UNIX commands (I'm like a scratched record!).

Code: Select all

sort A > A.sorted
sort B > B.sorted
comm -13 A.sorted B.sorted >> A
The solution sorts file A, then sorts file B and then appends to the existing A those that are in the sorted B that are not already in the sorted A.

Is this something along the lines of what you are looking for? If it is close but you want to do something like removing the records that are in A but not in B (i.e. the "old" records") then a variation on the comm command would be required. "man comm" for more details.

David.
(Previously known as D)

Be alturistic and donate your spare CPU cycles to research. http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1
Post Reply