upload big amount of files problem

Question

Hi
I have swing application that uploads very big number of files to ftp server. from 10k to 100k
I had a problem that sometimes files were not uploaded. I found that the ftp api that i use does not reuse the socket connections, so i tune the server to maxUserPort to 60k default wait time 3 minutes.
But if one client uploads 100k files this means that the client will run out of socket connections too? I suppose this!



I have the follwoing 2 exceptions in the log file of the swing application.

java.net.SocketOutputStream.socketWrite0(Native Method)
   java.net.SocketOutputStream.socketWrite(Unknown Source)
   java.net.SocketOutputStream.write(Unknown Source)
   sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
   sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)
   sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
   sun.nio.cs.StreamEncoder.flush(Unknown Source)
   java.io.OutputStreamWriter.flush(Unknown Source)
   com.enterprisedt.net.ftp.FTPControlSocket.writeCommand(FTPControlSocket.java:784)
   com.enterprisedt.net.ftp.FTPControlSocket.sendCommand(FTPControlSocket.java:765)
   com.enterprisedt.net.ftp.FTPClient.chdir(FTPClient.java:2747)
   admin.ProcessThread.uploadImagesToFTP(Unknown Source)
   admin.ProcessThread.doInBackground(Unknown Source)
   admin.ProcessThread.doInBackground(Unknown Source)
   javax.swing.SwingWorker$1.call(Unknown Source)
   java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   java.util.concurrent.FutureTask.run(Unknown Source)
   javax.swing.SwingWorker.run(Unknown Source)
   java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
   java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   java.lang.Thread.run(Unknown Source)
 
 //Method definition
 private native void socketWrite0(FileDescriptor fd, byte[] b, int off,
                 int len) throws IOException;
 
Obviously the exception is IO
 
com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:927)
   com.enterprisedt.net.ftp.FTPClient.initPut(FTPClient.java:1779)
   com.enterprisedt.net.ftp.FTPClient.putData(FTPClient.java:1838)
   com.enterprisedt.net.ftp.FTPClient.put(FTPClient.java:1595)
   com.enterprisedt.net.ftp.FTPClient.put(FTPClient.java:1580)
   com.enterprisedt.net.ftp.FTPClient.put(FTPClient.java:1558)
   admin.ProcessThread.uploadImagesToFTP(Unknown Source)
   admin.ProcessThread.doInBackground(Unknown Source)
   admin.ProcessThread.doInBackground(Unknown Source)
   javax.swing.SwingWorker$1.call(Unknown Source)
   java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   java.util.concurrent.FutureTask.run(Unknown Source)
   javax.swing.SwingWorker.run(Unknown Source)
   java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
   java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   java.lang.Thread.run(Unknown Source)
 
//Method definition
com.enterprisedt.net.ftp.FTPReply validateReply(java.lang.String string, java.lang.String string1) throws com.enterprisedt.net.ftp.FTPException { /* compiled code */ }
Obviously the exception is FTP

The program logic do not allow to upload batch of files or folders. It checks file by file if is uploaded and makes bulk inserts into database on some time.

Answer 1

Basically this won't work in FTP, as a new socket is required for each file transfers.

Common approaches include:

- zipping the files up and transferring the zip file => have a cron job unzipping them at the other end
- spacing the transfers out so that used sockets can be reclaimed by the operating system => probably not practical with this number of files
- use another protocol that doesn't have this flaw, e.g. SFTP.

Instead of checking file by file, do a directory listing to get a list of files, and upload a zipped bunch of files.

Answer 2

Why happens this problem, because of the ftp server (Gene6 ftp server) or because of the implemntation of your api?
Also how is a folder with files uploaded, is file by file or for the folder upload is used only one socket connection?
I upload gigabytes of images, i think that the zip/unzip operation may be slow. Also the unzipping of images must be instantly.

Is SFTP with your api free?

Answer 3

This is a consequence of how the FTP protocol is defined. It simply can't handle thousands of files very quickly. A folder upload requires a new socket for each file. That's why zipping works.

SFTP is not in the free client. Because it is encrypted it is also substantially slower than FTP.