DSXchange

there should be some way to limit the lookup data that is written to a hashed file.

e.g.

i extract 100 000 rows from a source (lets assume a subset of the customer base) and need to do a lookup on the latest order from a table in a different system with 10 000 000 rows (lets assume its the orders summary table). It would be beneficial to pass the list of account numbers into the job that creates the hashed file. This means that I only load 100 000 rows into the lookup, not records for each customer!

Limit in the SQL that extracts data to load in the hashed file.

uhhh, the lookup data is sourced from another system - i.e another database. Then the inline sub-query won't work.

Make it work. Load it into a work table in your database so you can do this - a pretty common technique, actually.

... or even a UV table in DataStage