Source Data Validation

[seethamsetty]

Hello All,

We have different applications feeding on to Dataware house. We need to do a pre validation and post-validation of source data before it is really processed by datastage jobs.

Pre-validation should include : Reading of file name , file size, file structure, and file content data format.

Post-Validation should include : Jobs processed status , No of rows rewriten etc.,

Will be thank full if any one throw some light on this , Is there any way that datastage can be used to do the above requirement.

or any one experienced this and tried to get some light on this. Please advise.

[keshav0307]

there are many ways that datastage can be used to do the requirement.
its very usual case nothing new.
how did you try to do achieve this.

[kduke]

You can use wc to get file size and number of rows. You easily write a routine to get these numbers. If you have fixed length records then it is a little more complicated.

The post stuff would need to use EtlStats or write your own routines or jobs to get the number of rows processed from the job. Do a search this has been covered a lot.

[ray.wurlod]

We need to do a pre validation and post-validation of source data before it is really processed by datastage jobs.

Why? What benefit will accrue? Are you just recording this information, or truly validating? What happens if a row in the file does not meet the specifications? What happens after that?