DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
faujong
Participant



Joined: 17 Aug 2018
Posts: 11

Points: 158

Post Posted: Fri Aug 17, 2018 10:16 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
I am using DataStage Designer 11.5.
I need to move a file to an archive folder, and add a timestamp in front of the file name. The current folder, archive folder and file name are parameters to the job.
#FileName_Parm# is MyFile_*.csv
#Directory_ParameterSet.$Dir_Parm# is /dev/DataStage/myProject/source
#Directory_ParameterSet.$Archive_Dir_Parm is /dev/DataStage/myProject/Archive

I use After-Job Soubroutine EXECSH command to move my file (with wildcard asterisk) to an archive folder:
Code:
mv #Directory_ParameterSet.$Dir_Parm#/TEST/#FileName_Parm# #Directory_ParameterSet.$Archive_Dir_Parm#/TEST/#FileName_Parm#


When running the above mv command, DataStage translates it to
Code:
mv /dev/DataStage/myProject/source/TEST/MyFile_*.csv /dev/DataStage/myProject/Archive/TEST/MyFile_*.csv


And, it moved MyFile_20180817.csv as My~1.csv instead of MyFile_20180817.csv

When I move the file to the archive folder, I also need to add a time stamp in the front of the file name, for example: MyFile_20180817.csv to 20180817_1057.MyFile_20180817.csv:
Code:
mv #Directory_ParameterSet.$Dir_Parm#/TEST/#FileName_Parm# #Directory_ParameterSet.$Archive_Dir_Parm#/TEST/`date +%Y%m%d_%H%M`.#FileName_Parm#


When running the above mv command, DataStage translates it to
Code:
mv /dev/DataStage/myProject/source/TEST/MyFile_*.csv /dev/DataStage/myProject/Archive/TEST/`date +%Y%m%d_%H%M`.MyFile_*.csv


And, it moved MyFile_20180817.csv as 201808~1.CSV instead of MyFile_20180817.csv

Thank you.
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42762
Location: Denver, CO
Points: 220350

Post Posted: Fri Aug 17, 2018 12:11 pm Reply with quote    Back to top    

At its heart, this really isn't a DataStage problem, per se. First issue is I see is you can't use wildcards on the target side for the move, nor should you mention the filename as the last argument is always the target folder - simply move it to the target folder. And I don't recall an ability using that command to rename the file as you move it to another location but it's been one of those weeks so others may need to confirm.

Code:
mv #Directory_ParameterSet.$Dir_Parm#/TEST/#FileName_Parm# #Directory_ParameterSet.$Archive_Dir_Parm#/TEST/

You also cannot use wildcards with the mv command to retain the existing filename, see one such discussion of what's going on here. It also notes a scripted loop can be leveraged or the find command or cp followed by rm, check out that part of the discussion as well.

Hope something in there helps,

_________________
-craig

Research shows that 6 out of 7 dwarves aren't happy
Rate this response:  
Not yet rated
faujong
Participant



Joined: 17 Aug 2018
Posts: 11

Points: 158

Post Posted: Fri Aug 17, 2018 2:42 pm Reply with quote    Back to top    

Thank you.
This code does move MyFile_*.csv to the Archive folder and the file name in the Archive folder is MyFile_20180817.csv.

Code:
mv #Directory_ParameterSet.$Dir_Parm#/TEST/#FileName_Parm# #Directory_ParameterSet.$Archive_Dir_Parm#/TEST/

Once the correct file is in the archive folder, can I rename/copy/move the file to include the timestamp ?
Can I combine 2 EXECSH in 1 Input Value for After-job subroutine ?
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42762
Location: Denver, CO
Points: 220350

Post Posted: Fri Aug 17, 2018 5:11 pm Reply with quote    Back to top    

Typically I would write a script so I have full control over exactly what happens when things go right and when things go wrong. However, if you want to string multiple commands together on one line, I would suggest using && as the separator rather than a semi-colon as the former only executes the next step if the previous step succeeds. The ; is unconditional which can lead to unintentional and potentially disastrous results.

_________________
-craig

Research shows that 6 out of 7 dwarves aren't happy
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42762
Location: Denver, CO
Points: 220350

Post Posted: Fri Aug 17, 2018 6:04 pm Reply with quote    Back to top    

Just to double-check, did you read the linked URL I posted earlier, specifically the here in blue up there? Not this one. Wink It answers your first "can I ?" question.

That or perhaps someone you work with who can help you with UNIX questions? The basics of this are nothing specific to DataStage, so don't let that stop you from reaching out to people that don't know anything about the tool if they can help with the command line stuff.

_________________
-craig

Research shows that 6 out of 7 dwarves aren't happy
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54395
Location: Sydney, Australia
Points: 295036

Post Posted: Sun Aug 19, 2018 6:40 pm Reply with quote    Back to top    

I'd also prefer a script. If you're constrained from using scripts (for example by short sighted political decisions), then you can construct a pipeline of commands to effect the move and subsequ ...

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
faujong
Participant



Joined: 17 Aug 2018
Posts: 11

Points: 158

Post Posted: Mon Aug 20, 2018 9:11 am Reply with quote    Back to top    

Thank you all.

This ExecSH command sucesfully moved the file to the archive folder, and add a timestamp in the front of the file name:

Code:
for file in /dev/DataStage/myProject/source/TEST/MyFile_*.csv ; do filename=`basename $file`; mv $file /dev/DataStage/myProject/archive/TEST/`date +%Y%m%d_%H%M`.${filename} ; done
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours