Directory listing encoding problem...

Question

Hi,

I've encountered an encoding problem in dir listing. After tracing the source code (edtFTPnet 1.2.4), I think I've found the cause.

Short description for the problem:
1. Get strange filenames listing for non-English (Traditional Chinese) filenames from FtpClient.DirDetails()
2. It only occurs on Windows FTP Servers (I am using Filezilla server on NTFS), which send out UTF-8 encoded directory listing.
3. Setting FtpClient.ContorlEncoding won't help (but works for UNIX-based servers).
4. Use Filezilla or Windows Explorer to browse the ftp server will get correct result.

After doing the following test, I've got the correct result:
1. ControlEncoding set to Encoding.GetEncoding(950);
2. Encoding for directory listing changed to UTF8 (source code change); originally, it uses Encoding.ContrlEncoding (see codes below).

It seems that encodings used for FTP commands and the result of directory listing doesn't always the same. It would be great f we can have another encoding setting, for example FtpClient.DirListingEncoding, for this.

How do you think?

                
In FTPClient.cs

                ...
                // send the retrieve command
                string command = full?"LIST ":"NLST ";
                
                ...
                ...

                FTPReply reply = control.SendCommand(command);
                ...
                ...
                string replyCode = lastValidReply.ReplyCode;
                if (!replyCode.Equals("450") && !replyCode.Equals("550") && !replyCode.Equals("226"))
                {
                    // get a character input stream to read data from
                    Encoding enc = controlEncoding == null ? Encoding.ASCII : controlEncoding;
                    //StreamReader input = new StreamReader(data.DataStream, enc);
                    // changed to UTF8 encoding and it will fix the issue
                    StreamReader input = new StreamReader(data.DataStream, Encoding.GetEncoding(65001)); // UTF8
                    ...
                    ...

Answer 1

Interesting. So there is no encoding that works for both control & directory listing?

Answer 2

Characters encoding is one of the most frustrating problems for me. When I first used edtFTPnet and saw the ControlEncoding and DataEncoding stuff, I knew you guys really have a clue on it and I really hope edtFTPnet can fully solve the problem. Thanks for the great work! edtFTPnet really rocks!


I did a further test and here is the result.

Test case:
Source:
Windows Vista English Edition
Default Code page for non-Unicode program: 1252 (English)
Filenames w/ Chinese characters are stored in NTFS (thus, these filenames are stored as Unicode)


Site1:
Linux Fedora 6 English Edition
EXT3
All default setting, thus, should be pure English.
Use FileZilla(runs on English Vista) to upload/download to: Won't work.

(a) Success!
FtpClient.ControlEncoding set to CP950 (Traditional Chinese)
Username: testuser, Password: 123456
FtpClient.Connect() --> OK
FtpClient.Login("testuser", "123456") --> OK
Get Dir Listing... --> Both English and Chinese filenames looks correct. OK.
Sucessfully transferred entire folders and sub-folders.


(b)
FtpClient.ControlEncoding set to UTF8
Username: testuser, Password: 123456
FtpClient.Connect() --> OK
FtpClient.Login("testuser", "123456") --> Exception: Please login with USER and PASS. (code=530)
Get Dir Listing... --> Don't have a chance to execute.



Site2:
Windows XP Pro Traiditioal Chinsese Edition
NTFS
Default Code page for non-Unicode program: 950 (Traditional Chinese)
FTP Server: FIlezilla Server
Use FileZilla(runs on English Vista) to upload/download: Works!


(a)
FtpClient.ControlEncoding set to CP950 (Traditional Chinese)
Username: testuser, Password: 123456
FtpClient.Connect() --> OK
FtpClient.Login("testuser", "123456") --> OK
Get Dir Listing... Chinese filenames NOT OK, got strange folder names. and when use the strange folder names to list sub-folders, exception: Directory not found. (code=550)


(b)
FtpClient.ControlEncoding set to UTF8
FtpClient.Connect() --> OK
FtpClient.Login("testuser", "123456") --> FAIL, exception: Syntax error, command unrecognized. (code=500)
Get Dir Listing... Don't have a chance to execute.


(c)
FtpClient.ControlEncoding set to CP1252 (English)
Manual change dir listing's encoding to UTF8
Username: testuser, Password: 123456
FtpClient.Connect() --> OK
FtpClient.Login("testuser", "123456") --> OK
Get Dir Listing... Engllish filenames OK, Chinese filenames NOT OK, but no exception. just received invliad filenames

(d) Success!
FtpClient.ControlEncoding set to CP950 (Traditional Chinese)
Manual change dir listing's encoding to UTF8
Username: testuser, Password: 123456
FtpClient.Connect() --> OK
FtpClient.Login("testuser", "123456") --> OK
Get Dir Listing... OK
Sucessfully transferred entire folders and sub-folders.


I am no FTP expert. Base on above tests, I guess FTP commands cannot be encoded as UTF8 right? And Filezilla server (and I guess most servers do) seems just return the orinial filenames in its origial encoding.


Not sure if I am providing useful information, please let me know if I can help to investigate the issue.