Moving file with wildcard and add timestamp to the file name

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
faujong
Participant
Posts: 11
Joined: Fri Aug 17, 2018 9:03 am

Moving file with wildcard and add timestamp to the file name

Post by faujong »

I am using DataStage Designer 11.5.
I need to move a file to an archive folder, and add a timestamp in front of the file name. The current folder, archive folder and file name are parameters to the job.
#FileName_Parm# is MyFile_*.csv
#Directory_ParameterSet.$Dir_Parm# is /dev/DataStage/myProject/source
#Directory_ParameterSet.$Archive_Dir_Parm is /dev/DataStage/myProject/Archive

I use After-Job Soubroutine EXECSH command to move my file (with wildcard asterisk) to an archive folder:

Code: Select all

mv #Directory_ParameterSet.$Dir_Parm#/TEST/#FileName_Parm# #Directory_ParameterSet.$Archive_Dir_Parm#/TEST/#FileName_Parm#
When running the above mv command, DataStage translates it to

Code: Select all

mv /dev/DataStage/myProject/source/TEST/MyFile_*.csv /dev/DataStage/myProject/Archive/TEST/MyFile_*.csv
And, it moved MyFile_20180817.csv as My~1.csv instead of MyFile_20180817.csv

When I move the file to the archive folder, I also need to add a time stamp in the front of the file name, for example: MyFile_20180817.csv to 20180817_1057.MyFile_20180817.csv:

Code: Select all

mv #Directory_ParameterSet.$Dir_Parm#/TEST/#FileName_Parm# #Directory_ParameterSet.$Archive_Dir_Parm#/TEST/`date +%Y%m%d_%H%M`.#FileName_Parm#
When running the above mv command, DataStage translates it to

Code: Select all

mv /dev/DataStage/myProject/source/TEST/MyFile_*.csv /dev/DataStage/myProject/Archive/TEST/`date +%Y%m%d_%H%M`.MyFile_*.csv
And, it moved MyFile_20180817.csv as 201808~1.CSV instead of MyFile_20180817.csv

Thank you.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

At its heart, this really isn't a DataStage problem, per se. First issue is I see is you can't use wildcards on the target side for the move, nor should you mention the filename as the last argument is always the target folder - simply move it to the target folder. And I don't recall an ability using that command to rename the file as you move it to another location but it's been one of those weeks so others may need to confirm.

Code: Select all

mv #Directory_ParameterSet.$Dir_Parm#/TEST/#FileName_Parm# #Directory_ParameterSet.$Archive_Dir_Parm#/TEST/
You also cannot use wildcards with the mv command to retain the existing filename, see one such discussion of what's going on here. It also notes a scripted loop can be leveraged or the find command or cp followed by rm, check out that part of the discussion as well.

Hope something in there helps,
-craig

"You can never have too many knives" -- Logan Nine Fingers
faujong
Participant
Posts: 11
Joined: Fri Aug 17, 2018 9:03 am

Post by faujong »

Thank you.
This code does move MyFile_*.csv to the Archive folder and the file name in the Archive folder is MyFile_20180817.csv.

Code: Select all

mv #Directory_ParameterSet.$Dir_Parm#/TEST/#FileName_Parm# #Directory_ParameterSet.$Archive_Dir_Parm#/TEST/
Once the correct file is in the archive folder, can I rename/copy/move the file to include the timestamp ?
Can I combine 2 EXECSH in 1 Input Value for After-job subroutine ?
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Typically I would write a script so I have full control over exactly what happens when things go right and when things go wrong. However, if you want to string multiple commands together on one line, I would suggest using && as the separator rather than a semi-colon as the former only executes the next step if the previous step succeeds. The ; is unconditional which can lead to unintentional and potentially disastrous results.
-craig

"You can never have too many knives" -- Logan Nine Fingers
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Just to double-check, did you read the linked URL I posted earlier, specifically the here in blue up there? Not this one. :wink: It answers your first "can I ?" question.

That or perhaps someone you work with who can help you with UNIX questions? The basics of this are nothing specific to DataStage, so don't let that stop you from reaching out to people that don't know anything about the tool if they can help with the command line stuff.
-craig

"You can never have too many knives" -- Logan Nine Fingers
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

I'd also prefer a script.

If you're constrained from using scripts (for example by short sighted political decisions), then you can construct a pipeline of commands to effect the move and subsequence rename in separate commands, probably making the second dependent upon the success of the first.
For example

Code: Select all

mv filename newdir && mv newdir/filename newdir/filename.`date +%s`
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
faujong
Participant
Posts: 11
Joined: Fri Aug 17, 2018 9:03 am

Post by faujong »

Thank you all.

This ExecSH command sucesfully moved the file to the archive folder, and add a timestamp in the front of the file name:

Code: Select all

for file in /dev/DataStage/myProject/source/TEST/MyFile_*.csv ; do filename=`basename $file`; mv $file /dev/DataStage/myProject/archive/TEST/`date +%Y%m%d_%H%M`.${filename} ; done
Post Reply