DSXchange

I have a file with Header, Trailer as first and last records. I know about the sequential file sed "1d;\$d" aspect but I was wondering if there was a way to do this in the FTP stage.

Any help would be great.

Thanks,

Scott

What's the next function after the FTP? If you're landing the file on the server, it's easier to edit the file with a script or with the file stage you are using.

If you are streaming the data directly to the next stage, you might find it better to do the editing there, a filter or transformer perhaps.

Thanks for the reply. My FTP stage (incoming) I have the URI and the needed options to view data. I then bring it to a transformer and I can use @INROWNUM for the first row but the LastRow() function isnt working.

Do the records have a unique identifier, which tells you what kind of record? If so, the transformer constraint can passively drop the header and trailer.

In the mainframe/COBOL world, record type is standard. If your file is not like that, you'll have to get creative with how to identify the records. Good luck.

What exactly is "not working" about the LastRow() function? The docs say that "the function returns TRUE if the current row is the last row in the data, or the end of wave" so wondering if the latter is tripping you up.

By the way - there are some positives and negatives about using the FTP stage. Unless it has changed in version 11, it does "row-by-row" transmissions with an acknowledgement request after each row.

This makes the FTP stage able to detect a transmission problem, but much slower than the unix sftp command.

The negative about the sftp command that ships with most UNIX operating systems is that it does not typically report transmission problems back to the calling process, especially when you use the interactive version of sftp.

Though sftp is much faster than the FTP stage, it can lead to problems with your process not receiving the complete file and not realizing it due to a lack of an error code. One solution is to buy a cheap third-party sftp tool that reports transmission errors (most do - they understand that it is a problem). The other is to either check file sizes on the source system or have a trailer record to validate "Yup - we got it all".

Since your file has a trailer record that can be used to confirm complete receipt, you might want to try using the UNIX sftp command. That would be significantly faster and would allow you to use commands to check for header / trailer and then strip them off before processing the file via sequential file stage. It could also report an error code when it doesn't find a trailer (or a file!) so the job doesn't attempt to process an old or partial file.

I used the FTP stage to upload a file then I used the external filter stage to use the sed '1d;$d'.

It worked.

Thanks all