FTP IOException and SocketException

Question

Hi,
I use your edtFTPnet/PRO dll. I make a secure connection to Host(Mainframe) and get approx. 100.000 files by FTP. The source is Host and the destination is a distributed server.
Tne connection part in my code is below:

ProFTPClient ftp = new ProFTPClient();
ftp.RemoteHost = FTPAddress;

ftp.LicenseOwner = Properties.Settings.Default.licenceOwner;
ftp.LicenseKey = Properties.Settings.Default.licenceKey;
ftp.ServerValidation = ProFTPClient.ServerValidationType.None;
ftp.ConnectMode = FTPConnectMode.ACTIVE;
ftp.Connect();
ftp.Auth(ProFTPClient.SecurityMechanism.SSL);
ftp.Prot(ProFTPClient.DataProtectionType.Clear);
ftp.Login(username, password);

ftp.DataEncoding = Encoding.GetEncoding("ISO-8859-9");
ftp.TransferType = FTPTransferType.ASCII;
ftp.Timeout = 60000;
FTPConnection.LogLevel = LogLevel.Debug;
FTPConnection.LogFile = logPathFTP;

This program runs once a week and sometimes we have connection errors that cuts FTP. The error types are below.

System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.24.96.92:64133
at EnterpriseDT.Net.Ftp.FTPClient.oJWL6OSmx(String , String )
at EnterpriseDT.Net.Ftp.FTPClient.Get(String localPath, String remoteFile)
at GenMetricCalculation.Utility.getProgramsByFTP(String genFolder, String FTPAddress, String username, String password)

System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.24.96.92:64957
at EnterpriseDT.Net.Ftp.FTPClient.oJWL6OSmx(String , String )
at EnterpriseDT.Net.Ftp.FTPClient.Get(String localPath, String remoteFile)
at GenMetricCalculation.Utility.getProgramsByFTP(String genFolder, String FTPAddress, String username, String password)


I read from your forums then I changed the connection type as ACTIVE and set the DEBUG level of the dll.
ftp.ConnectMode = FTPConnectMode.ACTIVE;

Then on my next try, I got the error below.

System.IO.IOException: An I/O exception occurred. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at Q9LMCYrADqjNKSv5HYL.KpQRy4rYp3EVwRHmEHA.Receive(Byte[] , Int32 , Int32 , SocketFlags )
at jQUkNFVqTfxJmq0G8qG.0iUmMqV3EAlMP92AG7a.Receive(Byte[] , Int32 , Int32 , SocketFlags )
at B74QFhVgUH3jEWnB3GO.r3neUHVykYpiq5HnJPi.Read(Byte[] , Int32 , Int32 )
--- End of inner exception stack trace ---
at EnterpriseDT.Net.Ftp.FTPClient.oJWL6OSmx(String , String )
at EnterpriseDT.Net.Ftp.FTPClient.Get(String localPath, String remoteFile)
at GenMetricCalculation.Utility.getProgramsByFTP(String genFolder, String FTPAddress, String username, String password)


The debug log of your dll says :

DEBUG [FTPClient] 13 Haz 2013 19:48:43.629 : Setting local path to \\copftpv11\Ftproot\CMDBTransfer\BecubicGenProgramCollection\DVL.GARIAG01.GENSRC\CAGM1481
DEBUG [ProFTPControlSocket] 13 Haz 2013 19:48:43.629 : Creating active data socket on port 0
DEBUG [ProFTPControlSocket] 13 Haz 2013 19:48:43.629 : Binding 10.129.1.127
DEBUG [FTPControlSocket] 13 Haz 2013 19:48:43.629 : ---> PORT 10,129,1,127,19,195
DEBUG [FTPControlSocket] 13 Haz 2013 19:48:43.629 : 200 Port request OK.
DEBUG [ProFTPActiveDataSocket] 13 Haz 2013 19:48:43.629 : Created active data-socket: Protocol=None, ProxyConnector=, CompatibilityFlags=0
DEBUG [FTPControlSocket] 13 Haz 2013 19:48:43.629 : ---> RETR CAGM1481
DEBUG [FTPControlSocket] 13 Haz 2013 19:48:43.660 : 125 Sending data set DVL.GAR$AG01.GENSRC(CAGM1481) FIXrecfm 80
DEBUG [ProFTPActiveDataSocket] 13 Haz 2013 19:48:43.660 : Awaiting a direct connection on active data-channel (timeout=60000ms)
DEBUG [ProFTPActiveDataSocket] 13 Haz 2013 19:48:43.660 : Accepted connection on active data-channel established.
DEBUG [SecureSocket] 13 Haz 2013 19:48:43.707 : Shutdown(Both)
DEBUG [ProFTPActiveDataSocket] 13 Haz 2013 19:48:43.707 : Attempt to close unconnected active data-channel.
DEBUG [FTPControlSocket] 13 Haz 2013 19:48:43.707 : 250 Transfer completed successfully.
DEBUG [FTPClient] 13 Haz 2013 19:48:43.723 : Setting local path to \\copftpv11\Ftproot\CMDBTransfer\BecubicGenProgramCollection\DVL.GARIAG01.GENSRC\CAGM1482
DEBUG [ProFTPControlSocket] 13 Haz 2013 19:48:43.738 : Creating active data socket on port 0
DEBUG [ProFTPControlSocket] 13 Haz 2013 19:48:43.738 : Binding 10.129.1.127
DEBUG [FTPControlSocket] 13 Haz 2013 19:48:43.738 : ---> PORT 10,129,1,127,19,196
DEBUG [FTPControlSocket] 13 Haz 2013 19:48:43.738 : 200 Port request OK.
DEBUG [ProFTPActiveDataSocket] 13 Haz 2013 19:48:43.738 : Created active data-socket: Protocol=None, ProxyConnector=, CompatibilityFlags=0
DEBUG [FTPControlSocket] 13 Haz 2013 19:48:43.738 : ---> RETR CAGM1482
DEBUG [FTPControlSocket] 13 Haz 2013 19:48:43.754 : 125 Sending data set DVL.GAR$AG01.GENSRC(CAGM1482) FIXrecfm 80
DEBUG [ProFTPActiveDataSocket] 13 Haz 2013 19:48:43.754 : Awaiting a direct connection on active data-channel (timeout=60000ms)
DEBUG [ProFTPActiveDataSocket] 13 Haz 2013 19:48:43.754 : Accepted connection on active data-channel established.
ERROR [SecureNetworkStream] 13 Haz 2013 19:49:44.332 : Socket.Receive() failed
ERROR [SecureNetworkStream] 13 Haz 2013 19:49:44.332 : System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
ERROR [SecureNetworkStream] 13 Haz 2013 19:49:44.332 : at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
ERROR [SecureNetworkStream] 13 Haz 2013 19:49:44.332 : at Q9LMCYrADqjNKSv5HYL.KpQRy4rYp3EVwRHmEHA.Receive(Byte[] , Int32 , Int32 , SocketFlags )
ERROR [SecureNetworkStream] 13 Haz 2013 19:49:44.332 : at jQUkNFVqTfxJmq0G8qG.0iUmMqV3EAlMP92AG7a.Receive(Byte[] , Int32 , Int32 , SocketFlags )
ERROR [SecureNetworkStream] 13 Haz 2013 19:49:44.332 : at B74QFhVgUH3jEWnB3GO.r3neUHVykYpiq5HnJPi.Read(Byte[] , Int32 , Int32 )
DEBUG [ProFTPActiveDataSocket] 13 Haz 2013 19:49:44.332 : Attempt to close unconnected active data-channel.
ERROR [FTPClient] 13 Haz 2013 19:49:44.332 : Caught exception
ERROR [FTPClient] 13 Haz 2013 19:49:44.332 : System.IO.IOException: An I/O exception occurred.
ERROR [FTPClient] 13 Haz 2013 19:49:44.332 : at B74QFhVgUH3jEWnB3GO.r3neUHVykYpiq5HnJPi.Read(Byte[] , Int32 , Int32 )
ERROR [FTPClient] 13 Haz 2013 19:49:44.332 : at System.IO.StreamReader.ReadBuffer()
ERROR [FTPClient] 13 Haz 2013 19:49:44.332 : at System.IO.StreamReader.ReadLine()
ERROR [FTPClient] 13 Haz 2013 19:49:44.332 : at EnterpriseDT.Net.Ftp.FTPClient.IZ9xqFoU3v(StreamReader input)
ERROR [FTPClient] 13 Haz 2013 19:49:44.332 : at EnterpriseDT.Net.Ftp.FTPClient.oJWL6OSmx(String , String )
ERROR [FTPClient] 13 Haz 2013 19:49:44.332 : CAUSED BY -

Answer 1

There is a limit to the number of files that can be transferred in a period of time - you might be exceeding it. Every transfer consumes one socket connection - and it takes time to free them up after closing. You may need to slow down your transfers.

Answer 2

There is a limit to the number of files that can be transferred in a period of time - you might be exceeding it. Every transfer consumes one socket connection - and it takes time to free them up after closing. You may need to slow down your transfers.

Hi,
Can you explanation how to slow down? Sleep a few seconds while getting files?
Also what is the reason that we sometimes live this problem and sometimes not?

Regards,

Answer 3

Yes - experiment with sleeping for a few seconds.

It's inconsistent because it depends on the size of the files - the smaller the files, the quicker the transfers - and the more sockets are consumed during a given period. There is a finite number of sockets available, and it takes a couple of minutes to free a socket after it is closed (see the TIME_WAIT state in TCP).

Answer 4

Yes - experiment with sleeping for a few seconds.

It's inconsistent because it depends on the size of the files - the smaller the files, the quicker the transfers - and the more sockets are consumed during a given period. There is a finite number of sockets available, and it takes a couple of minutes to free a socket after it is closed (see the TIME_WAIT state in TCP).

Thank you for your explanation.
How about FTP connection type ACTIVE or PASSIVE. Which one should I use?

Answer 5

How about FTP connection type ACTIVE or PASSIVE. Which one should I use?

It depends on your network environment. In active mode the server makes a new connection to the client for each file-transfer or directory-listing, whereas in passive mode the direction of the connections is reversed, i.e. the client connects to the server. Generally passive mode is more firewall-friendly.

- Hans (EnterpriseDT)

Answer 6

How about FTP connection type ACTIVE or PASSIVE. Which one should I use?

It depends on your network environment. In active mode the server makes a new connection to the client for each file-transfer or directory-listing, whereas in passive mode the direction of the connections is reversed, i.e. the client connects to the server. Generally passive mode is more firewall-friendly.

- Hans (EnterpriseDT)

Hi again,
There are 150.000 number of files. I cannot sleep 1-2 seconds. It should be in miliseconds. Can you recommend hom many miliseconds should the program wait?

Regards,

Answer 7

FTP is not really good at transferring lots of small files quickly since it tends to run out of ports. It needs a new port for each transfer. The OS reserves the port for up to two minutes after disconnect, so a port can't be reused within that time. Your sleep time therefore depends on the (1) the number of ports available for FTP on the server, (2) the duration that the OS reserves the port for after disconnect, and (3) the average time it takes to transfer each file. You can use those numbers to work it out for your situation.

Is using SFTP an option? It doesn't suffer from this problem since all files are transferred on one socket connection.

- Hans (EnterpriseDT)

Answer 8

FTP is not really good at transferring lots of small files quickly since it tends to run out of ports. It needs a new port for each transfer. The OS reserves the port for up to two minutes after disconnect, so a port can't be reused within that time. Your sleep time therefore depends on the (1) the number of ports available for FTP on the server, (2) the duration that the OS reserves the port for after disconnect, and (3) the average time it takes to transfer each file. You can use those numbers to work it out for your situation.

Is using SFTP an option? It doesn't suffer from this problem since all files are transferred on one socket connection.

- Hans (EnterpriseDT)

Hi,
Yes, we use SFTP. Then what changes?

Regards,

Answer 9

You have changed the code to use SFTP?

Answer 10

You have changed the code to use SFTP?

Hi,
The code was at the first message. We use secure FTP in out company. The code is as below.

ProFTPClient ftp = new ProFTPClient();
ftp.RemoteHost = FTPAddress;

ftp.************ = Properties.Settings.Default.licenceOwner;
ftp.********* = Properties.Settings.Default.licenceKey;
ftp.ServerValidation = ProFTPClient.ServerValidationType.None;
ftp.ConnectMode = FTPConnectMode.ACTIVE;
ftp.Connect();
ftp.Auth(ProFTPClient.SecurityMechanism.SSL);
ftp.Prot(ProFTPClient.DataProtectionType.Clear);
ftp.Login(username, password);

ftp.DataEncoding = Encoding.GetEncoding("ISO-8859-9");
ftp.TransferType = FTPTransferType.ASCII;
ftp.Timeout = 60000;
FTPConnection.LogLevel = LogLevel.Debug;
FTPConnection.LogFile = logPathFTP;