DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
ScottDun
Participant



Joined: 10 Dec 2015
Posts: 57

Points: 654

Post Posted: Tue Oct 17, 2017 7:57 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Windows
I have a file with Header, Trailer as first and last records. I know about the sequential file sed "1d;\$d" aspect but I was wondering if there was a way to do this in the FTP stage.

Any help would be great.

Thanks,

Scott

_________________
SCOTTDun
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 621
Location: Malvern, PA
Points: 5878

Post Posted: Tue Oct 17, 2017 8:27 am Reply with quote    Back to top    

What's the next function after the FTP? If you're landing the file on the server, it's easier to edit the file with a script or with the file stage you are using.

If you are streaming the data directly to the next stage, you might find it better to do the editing there, a filter or transformer perhaps.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596
Rate this response:  
Not yet rated
ScottDun
Participant



Joined: 10 Dec 2015
Posts: 57

Points: 654

Post Posted: Tue Oct 17, 2017 10:28 am Reply with quote    Back to top    

Thanks for the reply. My FTP stage (incoming) I have the URI and the needed options to view data. I then bring it to a transformer and I can use @INROWNUM for the first row but the LastRow() function isnt working.

_________________
SCOTTDun
Rate this response:  
Not yet rated
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 621
Location: Malvern, PA
Points: 5878

Post Posted: Tue Oct 17, 2017 10:36 am Reply with quote    Back to top    

Do the records have a unique identifier, which tells you what kind of record? If so, the transformer constraint can passively drop the header and trailer.

In the mainframe/COBOL world, record type is standard. If your file is not like that, you'll have to get creative with how to identify the records. Good luck.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42215
Location: Denver, CO
Points: 216749

Post Posted: Tue Oct 17, 2017 11:30 am Reply with quote    Back to top    

What exactly is "not working" about the LastRow() function? The docs say that "the function returns TRUE if the current row is the last row in the data, or the end of wave" so wondering if the latter is tripping you up.

_________________
-craig

Watch out where the huskies go and don't you eat that yellow snow
Rate this response:  
Not yet rated
asorrell
Site Admin

Group memberships:
Premium Members, DSXchange Team, Inner Circle, Server to Parallel Transition Group

Joined: 04 Apr 2003
Posts: 1636
Location: Colleyville, Texas
Points: 22245

Post Posted: Thu Oct 19, 2017 10:20 am Reply with quote    Back to top    

By the way - there are some positives and negatives about using the FTP stage. Unless it has changed in version 11, it does "row-by-row" transmissions with an acknowledgement request after each row.

This makes the FTP stage able to detect a transmission problem, but much slower than the unix sftp command.

The negative about the sftp command that ships with most UNIX operating systems is that it does not typically report transmission problems back to the calling process, especially when you use the interactive version of sftp.

Though sftp is much faster than the FTP stage, it can lead to problems with your process not receiving the complete file and not realizing it due to a lack of an error code. One solution is to buy a cheap third-party sftp tool that reports transmission errors (most do - they understand that it is a problem). The other is to either check file sizes on the source system or have a trailer record to validate "Yup - we got it all".

Since your file has a trailer record that can be used to confirm complete receipt, you might want to try using the UNIX sftp command. That would be significantly faster and would allow you to use commands to check for header / trailer and then strip them off before processing the file via sequential file stage. It could also report an error code when it doesn't find a trailer (or a file!) so the job doesn't attempt to process an old or partial file.

_________________
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2017
Rate this response:  
Not yet rated
ScottDun
Participant



Joined: 10 Dec 2015
Posts: 57

Points: 654

Post Posted: Thu Oct 19, 2017 10:39 am Reply with quote    Back to top    

I used the FTP stage to upload a file then I used the external filter stage to use the sed '1d;$d'.

It worked.

Thanks all

_________________
SCOTTDun
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours