Dynamic sequential file name generation

[ArndW]

I have a job which I am converting which creates an output sequential file "data.txt" and then uses an after-job OS exit to rename the file to "YYYYMMDD_data.txt".

I cannot change the parameters of the file, but would like to dispense with the external call for renaming, and though I'd come up with an elegant solution to the problem. I would insert a filter into the sequential file output stage to dynamically compute the current date and then tee the output to that dynamically named file.

The UNIX command "echo blah | tee /tmp/aw/`(date +"%Y%m%d")`_data.txt" works exactly as I'd like.

But when I put the "tee" command into the filter stage of my job, it doesn't expand the "`(date +"%Y%m%d")`" string. The same happens if I use "`$(date +"%Y%m%d")".

I've checked the output in compiled code and it does indeed put the in the "tee" command as I expected, but the date expansion isn't happening at runtime and I wonder if anyone has an idea on how I might be able to force that.

[ray.wurlod]

Do you need to escape the "%" signs?

[anbu]

Try this command in filter

Code: Select all

awk -v dt=$(date +"%Y%m%d") ' { print > dt"_data.txt" } '

[ArndW]

Ray - escaping the "%" doesn't change things.

Anbu - using awk doesn't work either, this time due to the single quotes, but the problem remains that the $(date...) function isn't being interpreted as a command.

I should have added in the original example, that the "tee /tmp/aw/$(date +"%Y%m%d")_data.txt" command results in the file "/tmp/aw/$(date" being created, which is expected if the command line isn't being expanded.

[Mike]

Hi Arnd,

I don't have the time to experiment, but I'm wondering if the eval command might force the date to get evaluated.

Something like:

Code: Select all

eval  "echo blah | tee /tmp/aw/`(date +"%Y%m%d")`_data.txt"

Mike

[qt_ky]

With the tee command, would you end up having two output files? If you have to remove one with an after-job subroutine, then it may defeat the purpose.

It's not taking any action based on the tick mark, possibly due to the way the filter command is run in a separately forked process. It's not evaluating the tick marks, so it finds the space after the date command and ends the path and file name.

Could you set an environment variable, such as DT, before the job runs, to the desired formatted value? Then reference it within the path and file name on the tee command (i.e. tee /path/$DT_data.txt or tee /path/$((DT))_data.txt).

[ArndW]

Eric - The tee command and file duplication isn't an issue. I am aware that my problem is that the command is being taken as a literal and not expanded as I'd expect it to be. I cannot change the environment or anything else outside of the job itself and want to replace an after-job subroutine which renames the file with something more elegant, hence my attempts at using the filter to create a dynamically named file.

[qt_ky]

I'm not sure if it can be done within one job in a more elegant way, short of working a support case. Escaping the tick marks with backslashes or more tick marks doesn't help. You could try more variations using single quotes, double quotes, or other special escape characters.

This may be less elegant than the after-job command, but could you insert a new job to work around the troublesome tick mark expansion problem in the filter command? For example, rename the current job, put a new job in its place (using the original job name), which calculates the date then calls the renamed job and passes a new file name parameter to the renamed job. The file name parameter could contain the current date.

That may be going outside your boundary by adding a job. A similar method would be to create a new sequence job that runs in parellel to the existing jobs, waits for data.txt, then renames it to include the current date, then loops (lather, rinse, repeat).

[ArndW]

Mike - I missed you post, sorry. I've played around with the options of the "eval" command for an hour with no luck, either. I think I might have to give up on this approach and stick with the after-job subroutine to rename the file after it gets created in the job. I was hoping for a more elegant solution.

[rameshrr3]

write to

Code: Select all

/dev/null

and in the filter command use

Code: Select all

sh -c "your big unix com mand"

. Im assuming you write to a server job seq file stage.

By the way did you escape the date command in backquote character

Code: Select all

 ` `

?
If you are using a filter stage in a parallel job , the moderator has to move your post there.

[ArndW]

rameshrr3 - The filter command you put in has the exact same problem that the original solution had, in that the "$(date)" command is not expanded but taken as a literal. See my original post, where I used backquotes to show that that approach doesn't work, either.

If you tried your recommended solution and it worked I'd love to hear more, as I cannot get it to work.

[qt_ky]

Workaround to opening a "Filter command" expansion (command substitution) support case:

Reusable shell script (i.e. tee_today.ksh):

Code: Select all

#! /bin/ksh
tee /data/file/path/or/path/parameter/`date +"%Y%m%d"`_data.txt

Filter command:

Code: Select all

/shell/script/path/or/path/parameter/tee_today.ksh

[ArndW]

qt_ky - thanks for that suggestion, but the whole point of this exercise was to avoid shelling out from the program and to remove an existing shell script (called as an after-job routine). So in this case it would be six of one or a half-dozen of the other.

[qt_ky]

You're welcome. Personally I think the elegant solution is to parameterize the filename to begin with (easy sequence job logic) and avoid renaming files post-job or creating two files via the tee command. The tee idea is certainly an elegant workaround given your constrained environment. However, it does not work as expected, which has also been confirmed by others.

This is a valid support case, should you choose to pursue it: "Command substitution fails within the Sequential File output stage's Filter command property." If you do, please share any findings. Good luck!