500MB file limit using GET()

Question

Everything works great, until I try to download a file in passv, ascii mode that is over 500MB. It downloads up to around 498MB then it just hangs there, endlessly.

Is this a known limitation?
Is there a work around?

I am using the latest version available here on the site.

Thank you very much in advance

Answer 1

Can't say we've tried a 500 MB file recently - 300 MB, yes.

It isn't a known limitation - do you have enough disk space?

Also, it isn't too common to transfer ASCII files that big ...

Everything works great, until I try to download a file in passv, ascii mode that is over 500MB. It downloads up to around 498MB then it just hangs there, endlessly.

Is this a known limitation?
Is there a work around?

I am using the latest version available here on the site.

Thank you very much in advance

Answer 2

I have 450GB+ free on a 600GB Drive.
What else should be checked?

ASCII Log files from media server are 500-3000MB routinely.

Cheers

Answer 3

I am having a very similar problem. I have implemented a multithreaded C# application that does the upload in a separate thread and then fires events to the main thread to update a progress bar, status bar, etc. It works perfectly for all smaller files I have tried (50k, 1 MB, 2MB, etc). I get this in my status window:

[06/13/2006 04:13:59] Setting up File Transfer.
[06/13/2006 04:13:59] Transferring file ... [from the Uploading event]
[06/13/2006 04:15:25] Transfer Complete. [from the Uploaded event]

When I upload a 432 MB ASCII semicolon-delimited file, the status is updated regularly until it gets to the final stage - the progress bar reads 100%, it says it has transferred 442968.75 KB out of 442968.75 KB, but the Uploaded event never fires and my "ThreadComplete" event (which fires when the threaded method ends) never fires either. This also happened with a 180.37 MB ZIP file. I left the application running for several minutes after it got to the final byte to see if it would ever complete, but it never did.

I checked the remote file and it is the exact same size as my local file. I also got the MD5 hash of each file, and they are identical - so the FTP was completed, but the FTP component never completed and exited. There is still plenty of disk space (10+ GB) left on the server. I tried using Binary instead of ASCII transfer type, and it did the same thing.

The code that creates the thread is:

            FTPThread ftpThread = new FTPThread();

            ftpThread.BytesTransferred+=new BytesTransferredHandler(ftpThread_BytesTransferred);
            ftpThread.Uploaded += new FTPFileTransferEventHandler(ftpThread_Uploaded);
            ftpThread.ThreadComplete += new EventHandler(ftpThread_ThreadComplete);
            ftpThread.ExceptionOccurred += new FTPThread.ExceptionEventHandler(ftpThread_ExceptionOccurred);
            ftpThread.Uploading += new FTPFileTransferEventHandler(ftpThread_Uploading);
            
            threadFTP = new System.Threading.Thread(ftpThread.BeginUpload);
            threadFTP.Start();

The code for my threaded method is:

 
        public void BeginUpload()
        {
           try
            {
                // Set server and log-in properties
                ftpConn.ServerAddress = _host.ToString();
                ftpConn.ServerPort = _serverPort;
                ftpConn.UserName = _userName.ToString();
                ftpConn.Password = _password.ToString();

                // Connect, put, and close
                ftpConn.Connect();
                ftpConn.UploadFile(_fileToUpload, _remoteFileName, false);
            }
            catch (Exception ex)
            {
                ExceptionOccurred(this, ex);
            }
            finally
            {
                if (ftpConn != null && ftpConn.IsConnected)
                    ftpConn.Close();
            }

            ThreadComplete(this, new EventArgs());
        }

Answer 4

Can you post a small segment of the logfile? Disk space obviously isn't a problem. It is possible that the FTP server has timed out the connection, and so the acknowledgement of the transfer is never sent on the control channel.

Answer 5

Here is the log from the transfer of my 180.37 MB file. The last line showed up in the console as soon as the progress bar reached 100% and the status bar showed "Transferring 184702 KB (180.37 MB) of 184702 (180.37 MB)," but the Uploaded event never fired, and the ftpConn.UploadFile(_fileToUpload, _remoteFileName, false); line never completed to move on to the next statement (which was a log command to list all the files in the remote directory).

WARNING: 'edtftp.log.level' not found or invalid - logging switched off
INFO [DupeChecker.FTPThread] 14 Jun 2006 09:24:16.941 : Setting connection parameters
INFO [DupeChecker.FTPThread] 14 Jun 2006 09:24:16.941 : Setting TransferType to Binary
INFO [DupeChecker.FTPThread] 14 Jun 2006 09:24:16.941 : Connecting
DEBUG [EnterpriseDT.Net.Ftp.FTPClient] 14 Jun 2006 09:24:16.951 : Connecting to 192.168.xx.xx:21
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 09:24:16.961 : 220 xxxxxxxx Microsoft FTP Service (Version 5.0).
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 09:24:16.971 : ---> USER Anonymous
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 09:24:16.971 : 331 Anonymous access allowed, send identity (e-mail name) as password.
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 09:24:16.971 : ---> PASS ********
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 09:24:16.971 : 230 Anonymous user logged in.
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 09:24:16.971 : ---> TYPE I
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 09:24:16.971 : 200 Type set to I.
INFO [DupeChecker.FTPThread] 14 Jun 2006 09:24:16.981 : Uploading file
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 09:24:17.082 : ---> PASV
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 09:24:17.082 : 227 Entering Passive Mode (192,168,xx,xx,19,50).
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 09:24:17.152 : ---> STOR Codes.txt
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 09:24:17.152 : 125 Data connection already open; Transfer starting.
The thread 0x1710 has exited with code 0 (0x0).
The thread 0x12ec has exited with code 0 (0x0).
The thread '<No Name>' (0xfdc) has exited with code 0 (0x0).
DEBUG [EnterpriseDT.Net.Ftp.FTPClient] 14 Jun 2006 09:49:37.162 : Transferred 189134859 bytes to remote host

For a smaller file (1.3 MB), this is what the log file shows after it transferrs all the bytes. The 226 Transfer Complete line that appears here never showed up for the larger files.

DEBUG [EnterpriseDT.Net.Ftp.FTPClient] 14 Jun 2006 10:14:09.977 : Transferred 1382400 bytes to remote host
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 10:14:09.987 : 226 Transfer complete.
DEBUG [DupeChecker.FTPThread] 14 Jun 2006 10:14:10.488 : Directory after put
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 10:14:10.488 : ---> PASV
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 10:14:10.488 : 227 Entering Passive Mode (192,168,xx,xx,4,80).
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 10:14:10.488 : ---> NLST .
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 10:14:10.488 : 125 Data connection already open; Transfer starting.
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 10:14:10.828 : 226 Transfer complete.
DEBUG [EnterpriseDT.Net.Ftp.FTPClient] 14 Jun 2006 10:14:10.828 : Found 10 listing lines
... (files are listed out in the output window) ...
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 10:14:10.828 : ---> QUIT
DEBUG [EnterpriseDT.Net.Ftp.FTPControlSocket] 14 Jun 2006 10:14:10.828 : 221  

Answer 6

It looks like the connection may have timed out, i.e. the server has terminated the control channel even though the file is still being transferred.

There are a number of ways we could deal with this - if you would like to try out a modification that may help, please email support at enterprisedt dot com and we'll send you a modified DLL to give a go.

Answer 7

Here is something that worked for me to keep the control connection alive. With this code you can transfer large files and still get the events to fire when the transfer is complete. You can try it out to see if it works for you, or you can wait until a new version of edtFTPnet is released with the ServerWakeupInterval functionality.

/// <summary> Default second interval for keep alive commands in seconds</summary>
private const int DEFAULT_KEEPALIVE_INTERVAL = 20;

/// <summary> seconds between keep alive commands are issued</summary>
private long keepAliveInterval = DEFAULT_KEEPALIVE_INTERVAL;
      
/// <summary>the last time a keep alive command was issued</summary>
private DateTime keepAliveLastIssued = DateTime.Now;

private void KeepAlive()
{
    if (((DateTime.Now - keepAliveLastIssued)).TotalSeconds >= keepAliveInterval)
    {
       keepAliveLastIssued = DateTime.Now;
      log.Debug("Sending Keep Alive after " + keepAliveInterval + " seconds." );
      control.WriteCommand("NOOP");
    }
}

And then call the KeepAlive() function from inside each PutBinary() or PutASCII() function before the check for the BytesTransferred event:

while ((line = input.ReadLine()) != null && !cancelTransfer)
{
   size += line.Length;
   monitorCount += line.Length;
   output.Write(line);
   output.Write(FTPControlSocket.EOL);

   KeepAlive();
   
   if (BytesTransferred != null && monitorCount > monitorInterval)
   {
      BytesTransferred(this, new BytesTransferredEventArgs(remoteFile, size));
      monitorCount = 0;
   }
}