hi,
I need to look for the existence of a file with current date in the name.
And then pass that name information to a data stage job.
Can you help?
Thanks
Rick
Read a file name from a unix server in Data Stage
Moderators: chulett, rschirm, roy
-
ep_datastage
- Participant
- Posts: 24
- Joined: Wed Apr 16, 2014 11:11 am
- Location: United States
Let me ask this up front - is this Unix server where DataStage is installed? If no, that complicates the answer.
Now, a typical solution would be a Sequence job with a Wait For File stage before the Job Activity stage. However, the WFF stage (unfortunately) doesn't support wildcards so the "current date in the filename" aspect of this shoots down use of that stage.
We'd use a shell script back in the day for this functionality. Basically poll for the existence of the file inside a loop every X until either 'too much time' had passed (whatever that would mean in each situation) in which case it would return an error, otherwise pass back the name of the found file. Sleep between checks unless you want to bring down the wrath of whomever is monitoring that server down on your head. Use an Execute Command stage to run it and be prepared to remove the Field Mark (@FM) characters from the answer before passing it along as a parameter to the Job Activity stage.
We much preferred the source send us an empty fixed name "go" file once the main file was transferred, essentially a semaphore to indicate we could process today's file. This way the WFF stage could be used and we would avoid getting any issues with large files, i.e. start processing them the moment they arrive but before they are "all in". This is especially handy when you expect multiple files.
As an FYI you could also leverage an Enterprise Scheduler to do most if not all of this front-end work if you have one.
Now, a typical solution would be a Sequence job with a Wait For File stage before the Job Activity stage. However, the WFF stage (unfortunately) doesn't support wildcards so the "current date in the filename" aspect of this shoots down use of that stage.
We'd use a shell script back in the day for this functionality. Basically poll for the existence of the file inside a loop every X until either 'too much time' had passed (whatever that would mean in each situation) in which case it would return an error, otherwise pass back the name of the found file. Sleep between checks unless you want to bring down the wrath of whomever is monitoring that server down on your head. Use an Execute Command stage to run it and be prepared to remove the Field Mark (@FM) characters from the answer before passing it along as a parameter to the Job Activity stage.
We much preferred the source send us an empty fixed name "go" file once the main file was transferred, essentially a semaphore to indicate we could process today's file. This way the WFF stage could be used and we would avoid getting any issues with large files, i.e. start processing them the moment they arrive but before they are "all in". This is especially handy when you expect multiple files.
As an FYI you could also leverage an Enterprise Scheduler to do most if not all of this front-end work if you have one.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
