Remove the First and Last Record of File in FTP

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
ScottDun
Participant
Posts: 61
Joined: Thu Dec 10, 2015 9:51 am

Remove the First and Last Record of File in FTP

Post by ScottDun »

I have a file with Header, Trailer as first and last records. I know about the sequential file sed "1d;\$d" aspect but I was wondering if there was a way to do this in the FTP stage.

Any help would be great.

Thanks,

Scott
SCOTTDun
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

What's the next function after the FTP? If you're landing the file on the server, it's easier to edit the file with a script or with the file stage you are using.

If you are streaming the data directly to the next stage, you might find it better to do the editing there, a filter or transformer perhaps.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
ScottDun
Participant
Posts: 61
Joined: Thu Dec 10, 2015 9:51 am

Post by ScottDun »

Thanks for the reply. My FTP stage (incoming) I have the URI and the needed options to view data. I then bring it to a transformer and I can use @INROWNUM for the first row but the LastRow() function isnt working.
SCOTTDun
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

Do the records have a unique identifier, which tells you what kind of record? If so, the transformer constraint can passively drop the header and trailer.

In the mainframe/COBOL world, record type is standard. If your file is not like that, you'll have to get creative with how to identify the records. Good luck.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

What exactly is "not working" about the LastRow() function? The docs say that "the function returns TRUE if the current row is the last row in the data, or the end of wave" so wondering if the latter is tripping you up.
-craig

"You can never have too many knives" -- Logan Nine Fingers
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

By the way - there are some positives and negatives about using the FTP stage. Unless it has changed in version 11, it does "row-by-row" transmissions with an acknowledgement request after each row.

This makes the FTP stage able to detect a transmission problem, but much slower than the unix sftp command.

The negative about the sftp command that ships with most UNIX operating systems is that it does not typically report transmission problems back to the calling process, especially when you use the interactive version of sftp.

Though sftp is much faster than the FTP stage, it can lead to problems with your process not receiving the complete file and not realizing it due to a lack of an error code. One solution is to buy a cheap third-party sftp tool that reports transmission errors (most do - they understand that it is a problem). The other is to either check file sizes on the source system or have a trailer record to validate "Yup - we got it all".

Since your file has a trailer record that can be used to confirm complete receipt, you might want to try using the UNIX sftp command. That would be significantly faster and would allow you to use commands to check for header / trailer and then strip them off before processing the file via sequential file stage. It could also report an error code when it doesn't find a trailer (or a file!) so the job doesn't attempt to process an old or partial file.
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
ScottDun
Participant
Posts: 61
Joined: Thu Dec 10, 2015 9:51 am

Post by ScottDun »

I used the FTP stage to upload a file then I used the external filter stage to use the sed '1d;$d'.

It worked.

Thanks all
SCOTTDun
Post Reply