We have a job that reads from external source stage.
The source program given in this stage is
"sh filename".
The volume of souce data is 193 million rows.
The job hangs after reading 17 million rows.
Is there any limitation for External source stage?
If this stage is replaced with sequential file stage, the job finishes successfully.
So use a Sequential File stage then. What purpose is the External Source stage serving here? "sh filename" is fairly meaningless as a source filename... is that "filename" a script?
-craig
"You can never have too many knives" -- Logan Nine Fingers
So... a script. And not a lot of other information to go on. I'm not sure how anyone can answer your question without seeing exactly what the script does and thus what "to get data from different files" means. At this point we don't even know if the ESS is what is "hanging" for you or even where you are attempting to write all of these records to. Perhaps your target is what hangs.
If this stage is replaced with sequential file stage, the job finishes successfully.
After processing how many files / loading how many records? All 193M or a small portion of the total?
-craig
"You can never have too many knives" -- Logan Nine Fingers
Look at your process sizes. Are they growing too big. Have you hit the OS limit for process size. I would think based on your source. You are hitting some kind of pipe limit. Output your file and read from it.
