Batch process using Oracle connector

dinakaran_s · Post by **dinakaran_s** » Thu Mar 15, 2012 8:09 pm

Hi,

Currently we are trying to fetch 10 million records from oracle (it may grow) and to creating a flat file. In between, and need to get the description from the master using lookup.

The jobs is running fine without any problem.

But the problem is, the scratch disk is getting full (20 gb) for just 1.9 million record. It is trying to pull all the records and creating multiple datasets. This is occupying most of the scratch disk space.

Is there any option in oracle connector to process the data in batches.

qt_ky · Post by **qt_ky** » Thu Mar 15, 2012 9:00 pm

What is the job design like? Does it have a Lookup stage with a normal lookup on a large number of records on the reference link?

ray.wurlod · Post by **ray.wurlod** » Thu Mar 15, 2012 9:43 pm

How much scratch space do you have? Can you allocate more?

How large are these 1.9 million records?