Our Products:   CompleteFTP  edtFTPnet/Free  edtFTPnet/PRO  edtFTPj/Free  edtFTPj/PRO
Ask a Question

Several issues with the multi-protocol client

0 votes
1.4k views
in Java FTP by (290 points)

I'm using the SecureFileTransferClient class because I need to write a Java client that supports both SFTP & FTPS (implicit & explicit), & this seemed like exactly what I need.  I'm using the latest pro version 5.2.5.  Here are my questions:

  1. I would like to use the new function of specifying a minimum TLS version when using FTPS.  I can see the new method takes an int, but there is no online doc that explains what that int should be if I only want to accept TLSv1.1 or higher.  Can you tell me what that int should be?

  2. Your online doc for using implicit FTPS says to do this:

    ftp.setImplicitFTPS(true);
    ftp.connect();
    ftp.auth(SSLFTPClient.PROT_PRIVATE);

    But how do I translate that series of commands when using the SecureFileTransferClient, which has no auth() method?  Heres what my code looks like so far:

    SecureFileTransferClient client=new SecureFileTransferClient();
    client.setProtocol(Protocol.FTPS_IMPLICIT);
    client.getAdvancedFTPSettings().setConnectMode(FTPConnectMode.PASV);
    client.getAdvancedSSLSettings().setSecurityMechanism(SSLFTPSecurityMechanism.AUTH_TLS);
    client.getAdvancedSSLSettings().setUseUnencryptedCommands(false);
    client.setServerValidationEnabled(false);

    So with server validation disabled, I need to execute the following command (after I connect to the server) in order for a file upload to work:
    client.executeCommand("PROT P");

    Is this really the correct way to get the SecureFileTransferClient class to encrypt the data channel?

    If server validation is enabled, then even that executeCommand() from above doesnt work when I try to upload a file (the server tells me a 'PROT P' is required, even though I executed that command prior to running client.uploadFile()).
    What am I missing here as I try to get implicit FTPS working using the SecureFileTransferClient class?

  3. I'm having problems getting key-based auth to work when using SFTP.  The following sftp command uses a private key file to connect to a host without needing a password:
    sftp -i /users/wasadm/.ssh/id_wasadm_rsa foo@acme.com

    When I try to emulate that same command in Java, I get this error message:
    ERROR [TransportProtocolOutputStream] 3 Jul 2018 00:13:32.607 : sendMessage() failed: Socket closed (state=5)
    ERROR [ConnectTask] 3 Jul 2018 00:13:32.607 : 1:Connect[acme.com:22] failed : The host signature is invalid or the host key was not accepted!
    com.enterprisedt.net.ftp.ssh.SSHFTPKeyException: The host signature is invalid or the host key was not accepted!
    Here is my Java code:
    client.setProtocol(Protocol.SFTP);
    client.loadSSHServerValidation("/users/wasadm/.ssh/known_hosts");
    client.getAdvancedSSHSettings().setPrivateKeyFile("/users/wasadm/.ssh/id_wasadm_rsa");
    client.getAdvancedSSHSettings().setPortsInKnownHosts(true); //some hosts do not use the default port of 22
    client.getAdvancedSSHSettings().setAuthenticationType(SSHAuthenticationType.PUBLIC_KEY);

    Any idea why the same files used by the built-in sftp command do not work with your Java client?

1 Answer

0 votes
by (162k points)

1. See SSLFTPClient.SSLVersion.TLS_V1_1 (we'll amend the documentation as it is a bit hard to find).

2. You don't need to do anything other than this to set up implicit mode TLS. 

 SecureFileTransferClient client=new SecureFileTransferClient();
 client.setProtocol(Protocol.FTPS_IMPLICIT);
 client.setServerValidationEnabled(false);

The other commands are set internally.

3. I'm not sure. We'll need a debug log to find out, which is going a bit beyond what this forum is for. Please open a support ticket here.

by (290 points)
Thank you for your answer to #1, it worked perfectly!
Your answer to #2 however, is what I initially had, but it does not work.  Here is the verbose output from the server:

220-FileZilla Server 0.9.60 beta
220 Welcome to Acme FTPS
---> USER Acme
331 Password required for Acme
---> PASS ********
230 Logged on
---> TYPE I
200 Type set to I
---> PWD
257 "/" is current directory.
---> NOOP
200 OK
---> TYPE A
200 Type set to A
---> PWD
257 "/" is current directory.
---> CWD 45146,724,724
250 CWD successful. "/45146,724,724" is current directory.
---> CWD /
250 CWD successful. "/" is current directory.
---> NOOP
200 OK
---> CWD 45146,724,724
250 CWD successful. "/45146,724,724" is current directory.
---> PWD
257 "/45146,724,724" is current directory.

ERROR [FTPClient] 3 Jul 2018 14:56:14.785 : Failed to configure data socket:java.net.SocketTimeoutException: Read timed out
ERROR [FTPClient] 3 Jul 2018 14:56:14.787 : Caught and rethrowing exception in initPut() : PROT P required
com.enterprisedt.net.ftp.FTPException: 521 PROT P required
    at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1330)
    at com.enterprisedt.net.ftp.FTPClient.initPut(FTPClient.java:2862)

ERROR [UploadFileTask] 3 Jul 2018 14:56:14.789 : 4:Upload[e:\tmp\test.txt=>test1.out] failed : PROT P required
WARN [FTPTaskProcessor] 3 Jul 2018 14:56:14.790 : Exception thrown in callback: PROT P required : PROT P required
---> NOOP
200 OK
---> PWD
257 "/45146,724,724" is current directory.
Started upload: test1.out
---> PASV
227 Entering Passive Mode (10,30,2,134,255,102)
---> STOR test1.out
521 PROT P required
test1.out - 0
Completed upload: test1.out
---> QUIT


As you can see, nowhere do you execute a 'PROT P' command on my behalf, yet clearly the server is expecting that before I try uploading a file.  Thats why I said, if I explicitly run

client.executeCommand("PROT P");

after I connect, but before running client.uploadFile(), it will work!

As I explained above, the problem with this 'solution' is it doesn't work once I enable server validation.  I get several 'FTPEx 521 PROT P required', even though I explicitly executed that command just before doing the upload.

So how do I get SecureFileTransferClient to automatically execute a PROT P for me, given that this server requires it?
by (162k points)
No-one really knows whether PROT P should be used for implicit mode, as there's no specification. It looks like Filezilla server does, hence the requirement to send it (see https://trac.filezilla-project.org/ticket/2581)

Please show us the log for when server validation is enabled.
by (290 points)
Heres the log when I enable server validation.  You will see that the 'PROT P' command is successfully executed on the server, & yet it still complains that its needed when I try to do the upload!


220-FileZilla Server 0.9.60 beta
220 Welcome to Acme FTPS
---> USER Acme
331 Password required for Acme
---> PASS ********
230 Logged on
---> TYPE I
200 Type set to I
---> PWD
257 "/" is current directory.
220-FileZilla Server 0.9.60 beta
220 Welcome to Acme FTPS
---> USER Acme
331 Password required for Acme
---> PASS ********
230 Logged on
---> TYPE A
200 Type set to A
---> PWD
257 "/" is current directory.
---> PROT P
200 Protection level set to P
---> NOOP
200 OK
---> TYPE A
200 Type set to A
---> PWD
257 "/" is current directory.
---> CWD 45146,724,724
250 CWD successful. "/45146,724,724" is current directory.
---> CWD /
250 CWD successful. "/" is current directory.
---> NOOP
200 OK
---> CWD 45146,724,724
250 CWD successful. "/45146,724,724" is current directory.
---> PWD
257 "/45146,724,724" is current directory.

ERROR [FTPClient] 2 Jul 2018 17:39:51.745 : Failed to configure data socket:com.enterprisedt.net.puretls.SSLPrematureCloseException: Short read
ERROR [FTPClient] 2 Jul 2018 17:39:51.752 : Caught and rethrowing exception in initPut() : PROT P required
com.enterprisedt.net.ftp.FTPException: 521 PROT P required
        at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1330)
        at com.enterprisedt.net.ftp.FTPClient.initPut(FTPClient.java:2862)
        at com.enterprisedt.net.ftp.FTPClient.b(FTPClient.java:2927)
        at com.enterprisedt.net.ftp.FTPClient.a(FTPClient.java:2634)
        at com.enterprisedt.net.ftp.FTPClient.put(FTPClient.java:2602)
        at com.enterprisedt.net.ftp.async.internal.UploadFileTask.run(UploadFileTask.java:128)
        at com.enterprisedt.net.ftp.async.internal.FTPTaskProcessor$b.run(FTPTaskProcessor.java:593)


com.enterprisedt.net.ftp.FTPException: 521 PROT P required
        at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1330)
        at com.enterprisedt.net.ftp.FTPClient.initPut(FTPClient.java:2862)
        at com.enterprisedt.net.ftp.FTPClient.b(FTPClient.java:2927)
        at com.enterprisedt.net.ftp.FTPClient.a(FTPClient.java:2634)
        at com.enterprisedt.net.ftp.FTPClient.put(FTPClient.java:2602)
        at com.enterprisedt.net.ftp.async.internal.UploadFileTask.run(UploadFileTask.java:128)
        at com.enterprisedt.net.ftp.async.internal.FTPTaskProcessor$b.run(FTPTaskProcessor.java:593)
ERROR [UploadFileTask] 2 Jul 2018 17:39:51.781 : 6:Upload[/tmp/test.txt=>test5.out] failed : PROT P required
com.enterprisedt.net.ftp.FTPException: 521 PROT P required
        at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1330)
        at com.enterprisedt.net.ftp.FTPClient.initPut(FTPClient.java:2862)
        at com.enterprisedt.net.ftp.FTPClient.b(FTPClient.java:2927)
        at com.enterprisedt.net.ftp.FTPClient.a(FTPClient.java:2634)
        at com.enterprisedt.net.ftp.FTPClient.put(FTPClient.java:2602)
        at com.enterprisedt.net.ftp.async.internal.UploadFileTask.run(UploadFileTask.java:128)
        at com.enterprisedt.net.ftp.async.internal.FTPTaskProcessor$b.run(FTPTaskProcessor.java:593)


com.enterprisedt.net.ftp.FTPException: 521 PROT P required
        at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1330)
        at com.enterprisedt.net.ftp.FTPClient.initPut(FTPClient.java:2862)
        at com.enterprisedt.net.ftp.FTPClient.b(FTPClient.java:2927)
        at com.enterprisedt.net.ftp.FTPClient.a(FTPClient.java:2634)
        at com.enterprisedt.net.ftp.FTPClient.put(FTPClient.java:2602)
        at com.enterprisedt.net.ftp.async.internal.UploadFileTask.run(UploadFileTask.java:128)
        at com.enterprisedt.net.ftp.async.internal.FTPTaskProcessor$b.run(FTPTaskProcessor.java:593)
PROT P required
WARN [FTPTaskProcessor] 2 Jul 2018 17:39:51.792 : Exception thrown in callback: PROT P required : PROT P required
com.enterprisedt.net.ftp.FTPException: 521 PROT P required

WARN [FTPConnectionPool] 2 Jul 2018 17:39:52.044 : Failed to cleanly close connection : Connection timed out.
com.enterprisedt.net.ftp.FTPConnectionClosedException: 421 Connection timed out.
        at com.enterprisedt.net.ftp.FTPControlSocket.a(FTPControlSocket.java:1368)
        at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1318)
        at com.enterprisedt.net.ftp.FTPClient.quit(FTPClient.java:4515)
        at com.enterprisedt.net.ftp.async.internal.FTPConnectionPool.disconnect(FTPConnectionPool.java:465)
        at com.enterprisedt.net.ftp.async.internal.DisconnectTask.run(DisconnectTask.java:130)
        at com.enterprisedt.net.ftp.async.internal.FTPTaskProcessor$b.run(FTPTaskProcessor.java:593)

com.enterprisedt.net.ftp.FTPConnectionClosedException: 421 Connection timed out.
        at com.enterprisedt.net.ftp.FTPControlSocket.a(FTPControlSocket.java:1368)
        at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1318)
        at com.enterprisedt.net.ftp.FTPClient.quit(FTPClient.java:4515)
        at com.enterprisedt.net.ftp.async.internal.FTPConnectionPool.disconnect(FTPConnectionPool.java:465)
        at com.enterprisedt.net.ftp.async.internal.DisconnectTask.run(DisconnectTask.java:130)
        at com.enterprisedt.net.ftp.async.internal.FTPTaskProcessor$b.run(FTPTaskProcessor.java:593)
WARN [FTPClient] 2 Jul 2018 17:39:52.047 : cancelTransfer() called
---> NOOP
200 OK
---> CWD /45146,724,724
250 CWD successful. "/45146,724,724" is current directory.
---> PWD
257 "/45146,724,724" is current directory.
Started upload: test5.out
---> PASV
227 Entering Passive Mode (10,30,2,134,254,152)
---> STOR test5.out
---> NOOP
200 OK
---> NOOP
200 OK
---> NOOP
200 OK
---> NOOP
200 OK
521 PROT P required
test5.out - 0
Completed upload: test5.out
---> QUIT
221 Goodbye
by (162k points)
Try sending these two commands in succession:

PBSZ 0
PROT P
by (290 points)
Thanks, that worked perfectly!

Now I have another question about the client.loadSSLServerValidation(); method.
For the test above, I used a PEM file that contained only the cert for Acme.com.  But what I really want is to point to a file that contains all the certs from valid public CA's that we trust.  This file exists & is used by the builtin /usr/bin/ftp command without any issues.  When I point to this file (which contains 175 certs), I get a bunch of warnings, like this:

WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.513 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.517 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.519 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.532 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.539 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.552 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.577 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.583 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.593 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.598 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.603 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.603 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.611 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.631 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.679 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.683 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1
WARN [SSLFTPCertificateStore] 6 Jul 2018 16:43:55.694 : Failed to decode certificate: java.io.IOException: Unrecognized OID for key1.2.840.10045.2.1

I think there are 17 listed, all with the same key.  At first I thought the file format must not be to your liking, as it has a bunch of human-readable lines in it, along with:

   -----BEGIN CERTIFICATE-----
   ... first certificate ...
   -----END CERTIFICATE-----

So I removed all the 'comments' in the file, leaving nothing but 175 BEGIN....END sections, 1 right after another.  But I still get the exact same warnings I pasted above.

So does that mean I have 17 bad certs, that couldn't be read from the file, or since the key listed is exactly the same, is it only 1 cert that couldn't be read?

Either way, is it possible for me to determine which cert(s) its having trouble decoding?
by (162k points)
ECC certificates aren't supported, and it is probably these that are the issue. We've just patched edtFTPj/PRO to ignore them and log the error as they are very rarely used. Please contact us for the patch:
https://enterprisedt.atlassian.net/servicedesk/customer/portal/1
by (290 points)
When you say ignore them, are you talking about the Warnings I pasted above?  It seems to already being doing that, as the non-ECC certs appear to still be usable.  Is that not the case with the current version I have?
Also, it seems that ECC certs may become more prevalent, according to this site:
https://www.ssl247.com/kb/ssl-certificates/generalinformation/what-is-rsa-dsa-ecc

Do you have any plans to support them in the future?  If so, any idea when?
by (162k points)
We'll support them eventually, but we haven't encountered anyone using ECC certs yet.
by (290 points)
I have a question about this method: client.createDirectory()

If the path string I pass in contains multiple path components, like "incoming/tickets/ID123456/logs", and none of those path components exist on the server, will this method create all those paths for me, or do I need to loop through all the path components and call createDirectory() on each one?
by (162k points)
Can you please create this as a separate question? It should really be one issue per discussion thread.
asked Jul 11, 2018 in Java FTP by (290 points) Using the SecureFileTransferClient.createDirectory() method
by (290 points)
It seems that I may have spoken too soon when I said that executing a 'PBSZ 0' followed by a 'PROT P' fixed that implicit FTPS issue.  It fails about half the time, & I don't know what else to look at (other than server logs, which I've put in a request for).  In the following log output, you can clearly see that both those commands are successfully executed on the server, & yet I still get the 'PROT P required' error from the server.

220-FileZilla Server 0.9.60 beta
220 Welcome to Acme FTPS
---> USER Acme
331 Password required for acme
---> PASS ********
230 Logged on
---> TYPE A
200 Type set to A
---> PWD
257 "/" is current directory.
---> PBSZ 0
200 PBSZ=0
---> NOOP
200 OK
---> TYPE A
200 Type set to A
---> PROT P
200 Protection level set to P
---> NOOP
200 OK
---> PWD
257 "/" is current directory.
---> CWD 45146,724,724
250 CWD successful. "/45146,724,724" is current directory.
---> CWD /
250 CWD successful. "/" is current directory.
---> NOOP
200 OK
---> CWD 45146,724,724
250 CWD successful. "/45146,724,724" is current directory.
---> PWD
257 "/45146,724,724" is current directory.

ERROR [FTPClient] 12 Jul 2018 16:38:18.612 : Failed to configure data socket:java.net.SocketTimeoutException: Read timed out
ERROR [FTPClient] 12 Jul 2018 16:38:18.614 : Caught and rethrowing exception in initPut() : PROT P required
com.enterprisedt.net.ftp.FTPException: 521 PROT P required
        at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1330)
        at com.enterprisedt.net.ftp.FTPClient.initPut(FTPClient.java:2862)


Any ideas on what else to try?
asked Jul 19, 2018 in Java FTP by (290 points) Do you support ssh keys in the known_hosts file that use the ecdsa-sha2-nistp256 algorithm?

Categories

EnterpriseDT 2020
...