Higher TransferNotifyInterval lowers transfer speed

Question

I have experimented with changing the TransferNotifyInterval value and have run into a performance problem.

When the TransferNotifyInterval value is left at the default or another small value (e.g. 5000), the transfer speed is about 800 - 900 KB / sec. When I increase the TransferNotifyInterval (e.g. 25000, 50000, 100000), my transfer speed is dramatically reduced to 150 KB / sec. The higher speed reported with a low TransferNotifyInterval matches what I see when transferring the file in FileZilla. This is on a 100 MBPS LAN.

When the TransferNotifyInterval is very low (e.g. at the default), the CPU usage of my application is high. I increased the interval so that it would update my status bar less frequently, and that does lower the CPU usage, but the transfer speed is adversely affected.

Is this a bug, or is there something else going on that would cause this behavior?

Answer 1

This is unexpected. It is hard to imagine what could be causing this - with a higher TransferNotifyInterval the client is doing less. I guess it is possible that the network is getting swamped and degrading, but for a 100 MBPS LAN that seems extremely unlikely, given the transfer speeds involved.

Anyone else have some ideas? We'll have to run some tests with large files and report back.

Is this a bug, or is there something else going on that would cause this behavior?

Answer 2

I have just run a bunch of tests and edtFTPnet behaved exactly as I expected. The larger the TransferNotifyInterval, the faster the transfer. For example, with TransferNotifyInterval=10,000 a transfer would take around 3.5 seconds, whereas with TransferNotifyInterval=1,000,000 it would take around 1.5.

Of course, the time taken depends a lot on what you are doing in your BytesTransferred event-handler. In my case I was simply updating a TextBox. However, the fact remains that the lower TransferNotifyInterval is, the more often the event-handler will be called, and hence the longer the transfer will take.

A couple of things are worth noting when using edtFTPnet (as opposed to edtFTPnet/PRO).

Firstly, in a Windows Forms application, it generally only makes sense to use the BytesTransferred event if you are employing a worker-thread to do the transfer. This is because the event-handler will usually be used to provide visual feedback to the user. This can only be done on the GUI thread, hence the transfer operation itself must NOT run on the GUI thread. If it does then the GUI won't get an opportunity to update (unless you call Application.DoEvents() repeatedly of course, which is not pretty). The PRO version has asynchronous methods that make it possible to run things in the background without explicitly using worker-threads, thus saving you a fair bit of work.

Secondly, whether or not you are using a worker-thread, the transfer will wait while the BytesTransferred event-handler is executed. It's therefore important to make the code in the handler as quick as possible.

I know this doesn't directly answer your query, but I'm not sure what else we can do to help you work out what's going on. I'd be very interested to hear of any experiments you do yourself to explain this strange behaviour.

- Hans (EDT)

Answer 3

I am performing the actual transfer in a separate thread, and the BytesTransferred event is used to update a progress bar and x KB / total KB message in a status bar. I then call Application.DoEvents();. So there are some calculations to find out KB and MB transferred and current speed (averaged over the entire transfer), but I'm not doing any really heavy processing like writing to a database, file, or other I/O operations. I would have expected the transfer speed to increase, if anything, when increasing the TransferNotifyInterval.

I cut out everything but just a basic x of y status bar change, and I still saw the same behavior (by watching how quickly the bytes were transferred). The speed comes in bursts - it will go slowly for a second or two and then race through 1MB+ in under a second, then slow down for another second or two, then speed up for a couple of seconds.

                long lngBytesTransferred = e.ByteCount;
                long lngTotalBytes = CurrentFile.Length;

                lblStatus.Text = "Transferring: " + 
                    lngBytesTransferred.ToString() + " of " 
                    + lngTotalBytes.ToString();

                Application.DoEvents();

My original event is:

        private void ftpThread_BytesTransferred(Object sender, EnterpriseDT.Net.Ftp.BytesTransferredEventArgs e)
        {
            //InvokeRequired required compares the thread ID of the
            //calling thread to the thread ID of the creating thread.
            //If these threads are different, it returns true.
            if (this.InvokeRequired) 
            {
                BytesTransferred_Callback d = new BytesTransferred_Callback(ftpThread_BytesTransferred);
                this.Invoke(d, new object[] { sender, e });
            }
            else
            {
                double dblKBTransferred = e.ByteCount / 1024;
                double dblTotalKB = CurrentFile.Length / 1024;

                double dblMBTransferred = dblKBTransferred / 1024;
                double dblTotalMB = dblTotalKB / 1024;

                double SecondsElapsed = ((TimeSpan)DateTime.Now.Subtract(_transferStarted)).TotalSeconds;

                progressBar.Value = (int)((dblKBTransferred / dblTotalKB) * 100);
                
                string CurrentSpeed = (dblKBTransferred / SecondsElapsed).ToString("###,###,###");

                lblStatus.Text = "Transferring: " + 
                    dblKBTransferred.ToString() + " KB (" + 
                    dblMBTransferred.ToString("f") + " MB)" +
                    " of " + dblTotalKB.ToString() + " KB (" +
                    dblTotalMB.ToString("f") + " MB).";

                lblStatus2.Text = "Transfer Speed: " 
                    +  CurrentSpeed 
                    + " KB / sec";

                //Application.DoEvents();
            }
        }

Answer 4

Yes, I noticed this surging also, but I have no explanation for it. It could be the FTP server, the server OS, the network infrastructure, the client OS, the .NET framework, or edtFTPnet. Have you got any theories?

Apart from this surging though, are you still finding that a larger TransferNotifyInterval causes a slower transfer?

- Hans (EDT)

Answer 5

The only thing I could think of to explain the surging is the TransferBufferSize. I thought that the pause in the transfer might be when it had to load the next chunk of the file from disk. However, I changed the buffer size to various values ranging from small to large, and it didn't appear to make a difference in the overall speed or the number of pauses in the transfer.

Answer 6

I don't know if this is related to this same issue, but I just noticed another oddity.

I set the TransferNotifyInterval and TransferBufferSize to the same value (100 KB interval):

ftpConn.TransferNotifyInterval = 102400;
ftpConn.TransferBufferSize = 102400;

But then as I watch the application's status bar, it updates exactly double that, every 200 KB. It is going slowly enough that I can see the value as it changes from, e.g., 96200, 96400, 96600, 96800, 97000. These values are in KB (calculated in the BytesTransferred event as e.ByteCount / 1024;).

Answer 7

Yes we fixed that recently. The just-released version of edtFTPnet/PRO includes this fix and so will the next version of edtFTPnet/Free. Thanks for letting us know.

- Hans (EDT)

Answer 8

When experimenting some more with these two values I found that the higher the TransferBufferSize, the faster my transfer goes. When I pumped it up to 500 KB, my speed skyrocketed from avg ~40KB/sec to ~500KB/sec! Also, the BytesTransferred event seems to fire based on the TransferBufferSize instead of the TransferNotifyInterval. If I change only the TransferNotifyInterval, I see no change in the way the status bar updates, but if I change the TransferBufferSize then it changes the update speed (as well as the transfer speed):

ftpConn.TransferNotifyInterval = 512;
ftpConn.TransferBufferSize = 512000;

The BytesTransferred event only fired every 500 KB instead of every 1/2 KB but at speeds of 500KB/sec. 512000 seemed to be the optimal setting; changing to 1 MB TransferBufferSize didn't increase the speed any more.

I watched the file grow on the destination FTP server and it did appear to transfer much faster with a higher TransferBufferSize.

When I swapped the two values for TransferNotifyInterval and TransferBufferSize:

ftpConn.TransferNotifyInterval = 512000;
ftpConn.TransferBufferSize = 512;

It still updated only every 500 KB, but the speed dropped to 66 KB/sec.

I want to set the TransferBufferSize higher to get the higher speed, but I don't want the update to be so infrequent. Anyone have any ideas on how to accomplish this?

Looking at ftpClient.PutBinary(), I'm wondering if the reason that TransferNotifyInterval doesn't work for intervals less than TransferBufferSize because it loads the entire buffer size first and only then has a chance to fire the event. This might have to be rewritten as an asynchronous BinaryWrite using the BeginWrite() method and have event handlers to get notifications.

// get an output stream
output = new BinaryWriter(data.DataStream);

while ((count = input.Read(buf, 0, buf.Length)) > 0 && !cancelTransfer)
{
   output.Write(buf, 0, count);
   size += count;
   monitorCount += count;
   
   KeepAlive();
   
   if (BytesTransferred != null && monitorCount > monitorInterval)
   {
      BytesTransferred(this, new BytesTransferredEventArgs(remoteFile, size));
      monitorCount = 0;
   }
}

Answer 9

Yes, you're right. Transfers are only notified after a Read or Write, thus after TransferBufferSize bytes have been transferred. So TransferNotifyInterval will always be greater than TransferBufferSize (unless TransferBufferSize is large enough that a Read doesn't fill it - I'm not sure if this is likely to happen).

Looking at ftpClient.PutBinary(), I'm wondering if the reason that TransferNotifyInterval doesn't work for intervals less than TransferBufferSize because it loads the entire buffer size first and only then has a chance to fire the event. This might have to be rewritten as an asynchronous BinaryWrite using the BeginWrite() method and have event handlers to get notifications.