"Unexpected null reply received" when ftping large

Question

Hi...

I've run across a problem where we have some ftp transfers failing with an "Unexpected null reply received" error. The strange thing is that smaller files are transferred without error but larger files fail with this error. I don't know yet exactly where the cutoff is but the files that have been failing are up in the 11 GB+ size range.

I'm using edtFTPj v1.5.6 connecting using "active" mode, the destination ftp server is running Fedora Core 8 with vsftpd 2.0.5-19.

Does anyone have any ideas on how to resolve this? I would upgrade to a newer version of the library if there's a good chance of that resolving the problem but I didn't see anything obviously related in the release notes for newer versions.

Here's the actual stack trace:

java.io.IOException: Unexpected null reply received 
at com.enterprisedt.net.ftp.FTPControlSocket.readReply(FTPControlSocket.java:805) 
at com.enterprisedt.net.ftp.FTPClient.validateTransfer(FTPClient.java:1812) 
at com.enterprisedt.net.ftp.FTPClient.put(FTPClient.java:1786) 
at com.enterprisedt.net.ftp.FTPClient.put(FTPClient.java:1770) 
at com.enterprisedt.net.ftp.FTPClient.put(FTPClient.java:1748) 
at com.pkg.app.FtpTransporter.transportFile(FtpTransporter.java:135) 
at com.pkg.app.TransportWorker.work(TransportWorker.java:118) 
at com.pkg.app.TransportWorker$1.run(TransportWorker.java:57) 
at java.lang.Thread.run(Thread.java:619)

Thanks,
Corey

Answer 1

Most likely the server is timing out the control connection after x seconds, while the transfer is still occurring. See if you can increase the server timeout.

Answer 2

Hi...

Thanks for your response. I logged a transfer (using lftp) and your supposition appears to be correct:

> put BIG_FILE
---> TYPE I
<--- 200 Switching to Binary mode.
---> PASV
<--- 227 Entering Passive Mode (208,38,15,26,173,215)
---- Connecting data socket to (208.38.15.26) port 44503
---> STOR BIG_FILE
<--- 150 Ok to send data.
**** control-socket: Connection timed out
---- Closing data socket
---- Closing control socket
---- Resolving host address...
---- 1 address found
---- Connecting to ftp.site.com (208.38.15.26) port 21
<--- 220 Welcome to Site. To upload, please cd to pub directory.  You can make sub-directory before upload. To download, please cd to pub or its sub-directory as informed.  List of files "ls" is not allowed
---> FEAT
<--- 211-Features:
<---  MDTM
<---  PASV
<---  REST STREAM
<---  SIZE
<---  UTF8
<--- 211 End
---> OPTS UTF8 ON
<--- 200 Always in UTF8 mode.
---> USER ftp
<--- 331 Please specify the password.
---> PASS XXXX
<--- 230 Login successful.
---> CWD /pub/nexen
<--- 250 Directory successfully changed.
---> TYPE I
<--- 200 Switching to Binary mode.
---> SIZE BIG_FILE
<--- 213 8036448820
---> PASV
copy: put rolled back to 8036514356, seeking get accordingly
<--- 227 Entering Passive Mode (208,38,15,26,130,16)
---- Connecting data socket to (208.38.15.26) port 33296
---> REST 8036448820
<--- 350 Restart position accepted (8036448820).
---> STOR BIG_FILE
<--- 150 Ok to send data.
---- Closing data socket
<--- 226 File receive OK.
15302132828 bytes transferred in 14849 seconds (1006.4K/s)
---- Closing idle connection
---> QUIT
<--- 221 Goodbye.
---- Closing control socket

The problem is I don't have control over the server configuration so I need some way to work around this issue on the client side. As you can see from the log above lftp is detecting when the control connection times out and reconnects and resumes the transfer. How can I detect when the control connection times out so that I can do a similar thing in my code?

Thanks,
Corey

Answer 3

You just need to write some code to do the same thing as lftp - which is resuming the transfer.

Put a try/catch around the uploadFile, and if it fails with an exception, use uploadFile again but using WriteMode.RESUME. Or you could have a loop that does this, with a maximum retry count.

Answer 4

I will try that, thanks. On a related note is there any chance that a future version of the library will detect the fact that the control socket has timed out/been disconnected and provide a more appropriate error message in the exception that is produced such as "Control socket disconnected" rather than the "Unexpected null reply received" error message which is misleading in this case?

Thanks again,
Corey

Answer 5

Yes, we'll take a look at this.

Answer 6

Message in future versions will be "Control channel unexpectedly closed". Included in edtFTPj/PRO 1.5.7, released today, and will be in the next j/free release.