DSXchange

Perhaps Iam missing something somewhere... May be the Lookup might be the confusion. It should be Join (With cross product) Full outer. If Input.Start > Lookup.Start And Input.End<Lookup.End Then 1 Else 0 Wouldn't the above code results to 1 for the first scenario that you had given and 0 for second...

Do a range look up of the same data and mark the looked up records as duplicate and reject it. In the given example, the second record of the first set will get a match on the first record. And hence duplicate. That sounds quite complicated. Do you mean, look up both the start and end date against ...

Looking like, all the question are a "copy paste" from else where.
Subhani, did you try searching the forum on your topics?
It has been discussed zillion of times.
Start here.

SMS type language can be avoided too.

As specified, 0 is not a valid value for given Pattern.
Default the 0 to a valid date value such as 510101.

Do you need an Aggregator? Why not just Sort the Data based on Code and Date and use Remove Duplicate to retain First. (Or last, based on sorting order)

Use sequential file option to import the metadata, later use it in your DB stage. If thats what you are looking for.
Is the sequential file fill with actual data or Metadata?

User status and Return code is also other options to pass value to the subsequent jobs.

Why do you have to abort? Why can to collect those records and notify?

You can promote that particular warning to Fatal Error message using Message handler.
Or you can pre check the number or rows with the given file mask in unix and not to run the job at all.

Do a range look up of the same data and mark the looked up records as duplicate and reject it.
In the given example, the second record of the first set will get a match on the first record. And hence duplicate.

Looks like the job might run for 7Hrs if let run alone. Pls test that and post the result, so that to make sure, the bottle neck is not at the ETL server Overload. Now further analyse the job. Is the time going on more into Extract or Load or the incremental data capture. Most likely the third part....

Usage analysis will be effective only if you store the metadate of the table and use it in the job. If you happen to give custome build or User defined query, it wont help too.

Providing SAME partition would never creates this warning. But when you dont want the same partition as the previous stage?
When you end up in re-partitioning?

If you have stored the metadata you can search and locate it from Repositry with the help of Usage Analysis itself.