Failed a keylookup for record

Hongqing · Post by **Hongqing** » Tue Nov 23, 2004 6:28 pm

I have a foreign key lookup parallel job. I use the lookup file set to lookup for keys. However, the lookup failed after processing up to 50%. I got the massege: Failed a keylookup for record. Then it gives another msg: Operator's runLocally() failed.

Anyone out there have any idea what could cause this problem?

richdhan · Post by **richdhan** » Tue Nov 23, 2004 11:54 pm

Hi,

How do you know 50% is complete?

This problem might be due to
1. Keys not mentioned properly
2. Incorrect Data

In the lookup stage there are options to Abort, Continue, Drop and Reject if the record is not found. Use the reject option and send the rejected records into PEEK stage.

HTH
--Rich

Pride comes before a fall
Humility comes before honour

Hongqing · Post by **Hongqing** » Wed Nov 24, 2004 10:14 am

Because the log msg. says "50% processed".

I added the peek as you suggested. The job finished processing total of 1008 records with 231 rejected. The rejected records all have the code "O". I checked the lookup file set and it does contain "O" there. Do not know why the lookup on "O" failed. Any ideas/suggestions?

Thanks a lot.

Hongqing · Post by **Hongqing** » Wed Nov 24, 2004 11:37 am

This porblem has been resolved. It turns out that we played with the partition of the lookup file set and that caused the cause the code it's looking up for missing from the partition and failed the lookup. Basically, the safe way is not to parttition the lookup file set if you do not have to.

richdhan · Post by **richdhan** » Thu Nov 25, 2004 8:25 am

Hi,

Always have 2 confiuration files. One configuration file to run on single node and the other to run on multinode. Always run your jobs first on single node configuration file. If the run happens sucessfully run the jobs on multinode configuration file. If there are any problems you can be rest assured the problem might be due to some wrong partitioning.

BTW what partitioning did you use. I think you have to use entire partitioning for the lookup file set.

HTH
--Rich

Pride comes before a fall
Humility comes before honour

T42 · Post by **T42** » Mon Nov 29, 2004 10:25 pm

Hongqing wrote:Because the log msg. says "50% processed".

That message only show up on Sequential File load. No matter what you are doing down the stream, the Sequential File being 50% load could still mean that only 30% are being processed down the stream where your lookup stage is.

Please observe the stage names next to log messages.

Rich: Incorrect. You can either use entire, OR Hash, OR Mod, as long as the input stream is partitioned the same way. In fact, you can partition it ANY way you want, but the reliability of matching data goes down.

On 7.x, AVOID handling paritioning. Even when you think you got a problem, DO NOT MANIPULATE PARTITIONING UNTIL YOU ABSOLUTELY KNOW WHAT YOU ARE DOING. (If you don't know what APT_DUMP_SCORE does, do not handle partitioning on stages that already handle partitioning until you learn this nifty debugging feature.)

Case in point: I was off for a holiday, my clients weren't (crunchtime after all is fun, isn't it?) They mucked around with some jobs for a while trying to get it to work.

I came back, and they bitched about how nullability are screwing up data so they have to repartition something or another. I look at them, dumbfounded, and asked them to show me. Every damn thing they did, I showed them what was supposed to be done (Change nullability in Transformer? Handle it with a "IsNull()" statement), and so on forth. I removed the partitioning manipulation they've done, and viola! Everything works. And they CLAIMED to have done the same thing over the weekend, and it didn't work.

I guess the computer just need my magic fingers.