Is there a way in data stage to verify that the number of records you have read from the source is the same number of records loaded in the target?
I know you can check the log, but what I want to know is how to build it into data stage as a number of records read in the beginning ,to do your all your transformation , and at the end to ? the number records processed and if they do not match to what was read then to abort the run ?
I think it is possible but how?
Any help will be appreciated?
Thx
Jan Moolman
QA of loading
Moderators: chulett, rschirm, roy
-
jan_moolman
- Participant
- Posts: 6
- Joined: Thu Apr 04, 2002 4:36 am
- Location: South Africa
-
vmcburney
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
Put an output link in each of your transforms that leads to a flat reject file and make these all reject links at the transform constraints screen.
You now have a couple of options:
- you can either set your warning limit on the job to 1 so the job aborts as soon as a row is rejected by any transform.
- Keep the warning limit at 50 and finish the entire job, then run an after-job routine that attaches the job, finds the input link, counts the number of jobs processed, finds the output link and counts the number of rows there and raises an abort error if a different amount is found.
After an abort you can view the reject file from Designer to see what rows were not being passed through the transformer.
For an examples of routines that counts the rows processed by links go to the sdk sample routines such as UtilityGetRunJobInfo and UtilityAbortToLog.
regards
Vincent
You now have a couple of options:
- you can either set your warning limit on the job to 1 so the job aborts as soon as a row is rejected by any transform.
- Keep the warning limit at 50 and finish the entire job, then run an after-job routine that attaches the job, finds the input link, counts the number of jobs processed, finds the output link and counts the number of rows there and raises an abort error if a different amount is found.
After an abort you can view the reject file from Designer to see what rows were not being passed through the transformer.
For an examples of routines that counts the rows processed by links go to the sdk sample routines such as UtilityGetRunJobInfo and UtilityAbortToLog.
regards
Vincent