Hi,
we have a server job that shall read all filenames from a directory and put it into a database table.
Within the folder stage only one column for the filename ist defined.
This job aborts, when in the directory is a file larger then 500 MB or when all files together are bigger then 500 MB.
It does not matter, what I am doing within the job, without sorting, transform and only writing to an hashed file the folder can be a little bit bigger than when writing to db stage, but it aborts with uvmalloc() memory exceeded error.
If there the files within the directory are smaller than 500 MB jobs are working fine.
If I delete some entries in the 500 MB file, so that it is smaller, the job works.
I can rebiuld it in parallel with external source using ls command, then it works. But using parallel job is not likely, because of sevrer routines that are used.
First question: does the folder stage really read all file content first, even I only use one column for filename?
Second: is there a parameter that limits memory for reading files or for the folder stage?
Thanks,
Mario
Folder Stage seems to read file content, only need filenames
Moderators: chulett, rschirm, roy
Re: Folder Stage seems to read file content while only want
From what I recall and what you've documented - yes. If literally all you need are the filenames, I'd suggest an ExecSH command Before Job to list the filenames to a flat file and then use that file as your source in the job.mario_j wrote:does the folder stage really read all file content first, even I only use one column for filename?
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
Did you ever get this resolved? Stumbled back on this today and just wanted to add that you could also use the Filter option of the Sequential File stage to feed in just the filenames using something like an "ls -1 <pattern>" O/S command.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers