Too many sockets?

Question

I have been using the edtFTP.net library to implement FTP client functionality in multi-threaded Windows Service application. Its work well and I like it a lot. However, I am running into problems when FTPing lots of files in quick succession. I receive the following error:

"Only one usage of each socket address (protocol/network address/port) is normally permitted"

Using netstat I can see that I have loads of sockets in the TIME_WAIT state and I believe I am hitting the upper limit of connections that can be made. The easiest way to solve this is to reduce the number of concurrent threads the service uses and reduce the TIME_WAIT registry setting to its lowest permittable value of 30 secs.

However, from stepping through the code it seems that everytime I issue a Put command a new socket is created, used and then losed (enters the TIME_WAIT state). Surely this could easily become a tight bottleneck? Is it possible for the FTPClient to keep a socket open and reuse it over and over again for each Put until the session is closed?

Any help is much appreciated.

Thanks,

Mike.

Answer 1

Unfortunately the FTP protocol works in this way - by opening a new data connection for each data transfer. This can result in the problems you've experienced with TIME_WAIT if you are transferring many files quickly.

Depending on your application one possibility might be to zip up bunches of related files and send them in one transfer.

However, from stepping through the code it seems that everytime I issue a Put command a new socket is created, used and then losed (enters the TIME_WAIT state). Surely this could easily become a tight bottleneck? Is it possible for the FTPClient to keep a socket open and reuse it over and over again for each Put until the session is closed?

Any help is much appreciated.

Thanks,

Mike.

Answer 2

Thanks for the speedy reply.

I just dug a little deeper into some of the other posts in this forum and found this doc: http://www.enterprisedt.com/general/doc ... erview.pdf which helps a lot.

What I have noticed is that the error only appears (so far) when the service is installed on my PC and connects to the FTP also on my PC, hence compounding the socket problem.

Unfortunately, zipping is not an option for my app but copying to a local disk when you are on the same machine is, of course, an option - instead of FTPing.

Thanks again.

Mike.

Unfortunately the FTP protocol works in this way - by opening a new data connection for each data transfer. This can result in the problems you've experienced with TIME_WAIT if you are transferring many files quickly.

Depending on your application one possibility might be to zip up bunches of related files and send them in one transfer.

However, from stepping through the code it seems that everytime I issue a Put command a new socket is created, used and then losed (enters the TIME_WAIT state). Surely this could easily become a tight bottleneck? Is it possible for the FTPClient to keep a socket open and reuse it over and over again for each Put until the session is closed?

Any help is much appreciated.

Thanks,

Mike.

Answer 3

I was going to post a thread similar to this one regarding the same error...

I have the need to transfer thousands upon thousands of small text files, and before long I reach this point. I cannot modify the receiving end of the equation so archiving etc. is not an option.

Is there no workaround for this problem? I'm not sure where I can take things further in my usage scenario if this is just "the way things are".

Suggestions welcome!
Binary Jay

Answer 4

Thought: Why does the same operation involving thousands of files work using an FTP client like ws_ftp if this is a windows tcpip limitation?

Answer 5

Best thing to do is give it a try with a few thousand files and see what happens.

It will depend very much on the TIME_WAIT setting on the machine the FTP server is running on, and the number of file descriptors permitted for the server process.

I was going to post a thread similar to this one regarding the same error...

I have the need to transfer thousands upon thousands of small text files, and before long I reach this point. I cannot modify the receiving end of the equation so archiving etc. is not an option.

Is there no workaround for this problem? I'm not sure where I can take things further in my usage scenario if this is just "the way things are".

Suggestions welcome!
Binary Jay

Answer 6

My workaround for this issue (but is obviously a huge performance sacrifice) is to just stick a Thread.Sleep of a couple hundred ms into the Put() loop. This seems to keep the queue of sockets flowing fine in a pinch.