Lookup job aborts

Hope · Post by **Hope** » Fri Jun 26, 2009 7:21 am

I am having a lookup problem in my job. The reference lookup file contains about 30,000 records.This jobs works fine for few thousands.When the file size is increased the job is getting aborted.I tried to put the data from file to a table and did sparse looup.That didn't work too.
This is the error
LKP_SAP_No,0: Could not map table file "D:/IBM_DQS_Config/Dataset/lookuptable.20090626.rknphzd (size 1261666600 bytes)": Not enough space.
LKP_SAP_No,0: Error finalizing / saving table /D=/ISDS/dynLUT405668aa2ae1

Please advice.

ArndW · Post by **ArndW** » Fri Jun 26, 2009 7:23 am

You sem to have run out of disk space in your Dataset directory at runtime. Increase the space or use an additional mount point with space for your datasets.

ArndW · Post by **ArndW** » Fri Jun 26, 2009 7:58 am

How does "DEV" relate to "D:/IBM_DQS_Config/Dataset/"?

Hope · Post by **Hope** » Fri Jun 26, 2009 8:11 am

I mean to say that we have 70GB of space in D:/ .70GB of space includes all parent and child folders. Do we need to have space in the "Dataset" folder?. Please advice.

priyadarshikunal · Post by **priyadarshikunal** » Fri Jun 26, 2009 10:40 am

Hope wrote:I mean to say that we have 70GB of space in D:/ .70GB of space includes all parent and child folders. Do we need to have space in the "Dataset" folder?. Please advice.

Monitor space while the job is running.

priyadarshikunal · Post by **priyadarshikunal** » Fri Jun 26, 2009 10:44 am

also which drive is used as Scratch?

There are a lot of factors which can create such problems.

Other jobs/ other stages on the same job may fill the available space.

chulett · Post by **chulett** » Fri Jun 26, 2009 10:48 am

And just to address the last question, there is no such thing as space "in a folder", for Windows the space is managed at the drive letter level and is available to any folder on that drive letter.

siauchun84 · Post by **siauchun84** » Tue Jun 30, 2009 9:01 pm

May I know how many Node you are using for your job?

miwinter · Post by **miwinter** » Thu Jul 30, 2009 4:28 am

Hope - sounds like you are doing a database lookup, correct?

Leonie - can you confirm what kind of lookup you were doing too please (Was it sparse or normal? Was it to dataset or database?)

Mark

Leonie · Post by **Leonie** » Thu Jul 30, 2009 4:39 am

Hi Mark

All sources are datasets or sequential files, no databases.

Leonie

miwinter · Post by **miwinter** » Thu Jul 30, 2009 5:08 am

OK thanks Leonie, I was looking at the angle of this being related to the temp directory instead of related to dataset or scratch. Could you check what you have this (TMPDIR) set to and verify space there during run time?

Mark

ArndW · Post by **ArndW** » Thu Jul 30, 2009 5:17 am

Here is a tip to reduce the size of your lookup, perhaps then it will fit into the 512Mb block limit = remove the length specifications for VarChar fields.

ArndW · Post by **ArndW** » Thu Jul 30, 2009 5:18 am

Here is a tip to reduce the size of your lookup, perhaps then it will fit into the 512Mb block limit = remove the length specifications for VarChar fields.

miwinter · Post by **miwinter** » Thu Jul 30, 2009 5:20 am

Good point Arnd, same principle as with any dataset.

What is the 512mb limit though? What imposes this?

ArndW · Post by **ArndW** » Thu Jul 30, 2009 5:26 am

That limit is set in the executable and can be modified (with ldedit on AIX) but I am sure that there are drawbacks to doing so, otherwise the limit would be set higher in all jobs.