LookupFileSet size issue..

kura · Post by **kura** » Thu Sep 23, 2004 4:12 pm

hi ,

I am trying to create a LkP Fs..with about 21 million records..each of 36 bytes ..and my Job aborts saying that it has ran out of space after inserting 14 million records..now the resource space has about 254 GB free..

..the exact error msg is :

lkpfsWFccsiCISLPIDxref,0: Could not map table file "/opt/etl/staging/mad/resource/dataset/dataset1/lookuptable.20040923.cgqqvoa (size 1238384552 bytes)": Not enough space.

..any ideas ??

Thanks

chaitanyavm · Post by **chaitanyavm** » Thu Sep 30, 2004 11:36 am

Hi Kura,

You may be exceeding the file size limitation on the OS. Find out the max file size on your OS. Then divide that by the size of records and that will give you the size of the max file size. You may have to partiition the job as well. So that the file is within the OS limit.

Thanks
Chaitanya

kura · Post by **kura** » Thu Sep 30, 2004 4:50 pm

Hi,

Actually, it is not we could create dataset of size more 5GB. That too on each node.

Vijay

pavankvk · Post by **pavankvk** » Tue Oct 05, 2004 4:33 pm

check the ulimit size for the user who runs this job..also when u r handling that much data..it is advised to use a join instead of look up..look up doesnt give u performance..take it for granted