How does edt-ftp/j-pro handle 0D0D0A sequence in ascii mode?

Question

I'm seeing a discrepancy between what AIX's ftp command does and what EDT ftp/j-pro does when transferring a file from a Windows machine if the file contains the sequence 0D0D0A.  It looks like AIX removes both ODs while ftp/j only removes 1.  Is this what is actually happening?  If so is it a bug in ftp/j or in AIX?

EDIT:  My math was off yesterday.  It turns out that ftp/j-pro is not stripping ANYTHING from the sequence 0d0d0a.  Thus the file on the target system still contains the 0d0a sequence (albeit with a 0d prefix).

EDIT 2 (Last edit):  I've been able to determine the reason for this character sequence.  The file is laid out as records with each record starting on a new line.  However, records can be long enough that they have to be broken up into multiple lines so whatever is generating the records is adding 0D as a record delimiter.  Since it is generated on a Windows machine each record ends with <end-of-record><CR><LF> ie 0D0D0A,

Answer 1

I think 0D0D0A isn't a standard character sequence for a text-file regardless of which platform you're on.  On Windows it could be interpreted as either one or two newlines and I think 0A isn't generally used in text-files on *NIX at all.  Since this character-sequence isn't defined you should expect various behaviours depending on the inclinations of the developers.  So I wouldn't really call it a bug in either platform.  It's more of a feature.  :-)

To avoid this discrepancy I'd probably either use binary transfer-mode or process the file prior to transfer.

Comments

Not quite the answer I was hoping for since both of those options are outside of my control but that is not your problem.  Does the new information change your answer at all?

---

FTP mandates that EOL=0d0a while transferring, so j/PRO doing an ASCII transfer from Windows should recognize the 0d0a sequence as an EOL, but since Windows has the same EOL definition as FTP it should leave it unchanged.  With respect to the single 0d, to be honest with you, I can't remember how we've implemented it, but from what you've said it appears that we've chosen to leave it as is since the standards don't tell us to do otherwise.

By the way, this is only part of the story.  As I said all ASCII FTP transfers are supposed to send 0d0a for EOLs while transferring, but it's up to the server to choose what it will do with that.  E.g. on *NIX 0d0a will get converted to 0a.  And like the client, if the server encounters an 'invalid' character sequence then it also has to make a choice.

You say that you have no control over the client.  If you have control over the server then perhaps you can do some postprocessing on the incoming files to deal with 0d0d0a?

---

First off, thank you for the fast responses.

For the record, we are providing the FTP client so we do have some control but the end user gets to choose the transfer mode as well as other FTP options.

So if I understand correctly, FTPj is handling EOLs differently based on the character that precedes the EOL?  That might not technically be a bug (I'm not convinced) but it does seem to be an non-intuitive way to interpret the spec.

Post-processing is an unpleasant option but it looks like that is what we'll have to do.

---

You're misstating situation.  On Windows there isn't a single character that represents EOL.  0d by itself isn't an 'EOL character' and 0a isn't either.  EOLs are specified by a sequence of two characters, 0d0a.  Our code looks for those two characters and when it encounters them it substitutes the FTP EOL sequence, which happens to be the same (on Windows), i.e. 0d0a.  When edtFTPj/PRO encounters 0d without a subsequent 0a character then it just treats it the same as any other character and sends it straight through the data channel.

Yes, we could have done either of two things instead, (1) ignore the single 0d, or (2) pretended that the single 0d was actually a 0d0a and treat it as an EOL.  But what we decided to do was to just send it through the data channel by itself.

That is the decision that was made somewhere between 1999 when edtFTPj was first released and now.  It's not wrong, because this situation is not covered by the spec.  It's simply the decision that we made.  If we were to change it to suit you then someone else would complain that the single 0d had been changed.

Having said that, you can purchase the source-code and change it to behave the way you want it.  Or, depending on how your code is written, you can probably also preprocess the file or stream before you pass it to edtFTPj/PRO.

---

By the way, the main issue here is that the file being transferred is not a valid Windows text file.  I think your focus should be on correcting that rather than changing the behaviour of downstream components.