451 "Broken pipe" when reading large directories with edtFTPj

Question

We've been using edtFTPj successfully for years. However, recently we ran into "451 Transfer aborted. Broken pipe" errors when performing "dir" on an FTP server. The reason was that the FTP directory contained an unusual amount of files (about 2000). We solved our problem on short term by temporarily moving away files from the FTP server and subsequently feeding them back in "digestable" chunks of 200.

We turned on tracing and reproduced the problem to find the cause of the "451 Transfer aborted. Broken pipe" errors. Client and server logs are provided here: http://pastebin.com/vL2YtDXb. According to our analysis, this is what happens:

1. edtFTPj successfully connects to the FTP server. It opens a control port (21) which will remain open for the entire FTP session. 
2. Our application wants to list the FTP directory, however at maximum 10 files. This is implemented like this:
   1. Application calls com.enterprisedt.net.ftp.FileTransferClient.directoryList() passing an own DirectoryListCallback implementation (DirCallback.java).
   2. edtFTPj sets off the DIR command and receives the directory contents over a data socket (cf. com.enterprisedt.net.ftp.FTPClient.dir())
   3. FTP server writes directory contents into the data socket, each entry is one line
   4. FTPClient.dir() reads the data socket line by line and uses the application's DirectoryListCallback implementation ("DirCallback") to process them (cf. while loop at FTPClient.java:3806)
      1. DirCallback calls DirectoryListArgument.abortListing() after the 10th entry; this makes FTPClient.dir() leave the while loop
      2. FTPClient.dir() closes the data socket and reads the FTP Reply code from the control socket, expecting one of 226 or 250 (FTPClient.java:3831 and 3837)
         ==> normally everything is fine and the server sends 226 or 250
         ==> but if there are "too many" files on the FTP server, it sends a 451 (Transfer aborted. Broken pipe). 

The 451 error seems to occur when ever edtFTPj closes the data socket before the server managed to write all the directory contents into the data stream. The larger the directory contents, the more likely the 451 error occurs. In fact, if we increase the maximum size of files to read from 10 to 1000, we can successfully read large directories.

==> A possible solution for this problem is to make FTPClient.dir() accept 451 (addtitionally to 226 and 250) as a valid code after closing the data socket (FTPClient:3835). Maybe this could be configurable. 

This feature is necessary, because the current design doesn't allow us to solve the problem ourselves: We could subclass FTPClient.java and override dir(), but we coundn't use it in FileTransferClient.java (private field "ftpClient" without setter).

 

Facts:

• FTP client: edtFTPj-2.4.0.jar (OS: AIX 6.1)
• FTP Server: ProFTPD 1.3.4b (maint) (OS: GNU/Linux x86_64))

Logs: http://pastebin.com/vL2YtDXb

 

Answer 1

So you only want 10 files from the listing? How do you know what files you want?

Comments

The order does not matter in this case.

The application polls the FTP server. The user can configure the polling interval and how many files should be read at maximum.

---

So you don't really care which filenames you get, as you just keep getting them?

---

Yes, that's true for edtFTPj's point of view. Our application will filter the desired files. E.g. if it wants to read "*.tmp" it will continue to read the DIR results until it has found 10  files ending with ".tmp". So precisely, edtFTPj may have to skip through more than 10 files in FTPClient.dir() before our DirCallback signals to abort further processing.

---

Have you tried using the filter in the dir()? Some servers support this, and that would reduce the number of file names returned.

---

Our application calls FileTransferClient.directoryList(String, DirectoryListCallback). I'm not aware of any other way to perform the directory listing.

However, even a filter wouldn't really solve our problem, because in our original production problem, we needed to process all files.

---

We are reluctant to modify the client to accept an error code, but we could modify  FileTransferClient to allow you to set the FTPClient so you could use a subclassed FTPClient. Would that help?

---

Yes, that would help us.

---

Can you please try that out by editing the current source? That'll save some to and fro'ing.

---

Sorry, I don't know how to do that.

---

I still think this issue should be solved within edtFTPj itself. Please consider my reasoning:
1. The FTP client is the one that is closing the data socket (FTPClient:3831).
2. At this point the FTP client does not know if the FTP server is still writing to the data socket or not. So it must anticipate both cases:
   Case 1: The FTP server finished writing to the data socket. -> Reply codes 226 / 250.
   Case 2: The FTP server runs into an IO error while still writing to the data socket. -> Reply code 451.
3. At this specific point the FTP client does not care if the FTP server was still writing to the data socket, because it purposly wants to abort processing the "DIR" results.
   ==> Hence, 451 should be accepted as a valid return code (only at this specific point). It should be logged as warning, however.

---

Why don't you try modifying the code and seeing if this works for you?  It's easy to build from the command line.

---

I was thinking a little more about this - why don't you just catch the exception and check if the error code is 451?

---

I only want to tolerate 451 if it occurs at FTPClient:3836, but not if it occurs elsewhere, e.g. at FTPClient:3790. So catching the FTPException won't really solve my problem.

I've managed to build edtFTPj. I edited it so that I can introduce an own, overridden FTPClient in FileTransferClient. However, subclassing and overriding FTPClient won't work, because the dir()-method accesses private members ("cancelTransfer" and "lister") and methods ("closeDataSocket()") that are not available to my subclassed FTPClient.
Even if if it were possible (e.g. by adding getters to FTPClient), the solution with subclassing and overriding has several drawbacks:
1. The solution duplicates a lot of code (dir() is about 90 line long!). Upgrading edtFTPj in future becomes error-prone, because we need to verify that the duplicated code in our FTPClient still matches the code in the new edtFTPj's FTPClient.
2. FTPClient.dir() is private, so it's necessary to overide all callers of dir() aswell (dir(String, boolean), dirDetails(String,DirectoryListCallback)) to call the overridden version of dir().
3. Every other user of edtFTPj potentially has the same problem slumbering when reading large directories with an aborting DirectoryListCallback.

Dealing with 451 in FTPClient directly would be a much simpler, sustainable solution.

---

You know when you've called abortListing, so if you get a 451 directly after this in the dir method, why can't you just ignore it?

---

Actually because we want separation of concerns in our design: On one side we have the application code that wants to list a directory by calling edtFTPj (FileTransferClient.directoryList()). On the other side wie have our DirectoryListCallback-implementation which is called by edtFTPj and deals with processing directory results. The first part should not need any knowledge about how directory processing occurs.

However, I admit that it is a possible solution to try/catch the FTPException in our application, check if it is a 451 and ask our DirectoryListCallback-implementation if it requested to abort. If both is true, the application can continue. Thanks for that idea. I'll try that.

---

The try/catch solution worked for us. Thank you.

I still think this is a workaround solution. Really this case should be handled by edtFTPj internally, because edtFTPj is prematurely closing the FTP data socket.