Is it possible to download large files in chunks?

Question

I have tried DownloadByteArray() and DownloadStream() for downloading binary files of ~450 MByte. This generates an OutOfMemory exception. I am positive the exception is thrown from the Download... method.

My wish is to establish a download stream to pour data into a zip output stream without needing to worry about file size.

My assembly is 32 bit.

As I understand the API, asynchronous use does not necessarily solve my problem.

Answer 1

DownloadByteArray reads all data into a byte[] before returning, so it doesn't work well for very large downloads.  DownloadStream simply sends all data to the stream object that's passed to it.  I understand you're passing in a stream that is capable of writing to ZIP files.  If you're getting an out-of-memory error then it's probably because that stream doesn't flush the data to the disk as it's being received and is therefore buffering the whole lot.  If this is the case then you either need to make changes so that the stream automatically flushes the data or else write your own stream class that takes control of this.  You may find the System.ServiceModel.Channels.DelegatingStream class useful as it essentially allows you to add functionality to an existing stream class by selectively overriding particular methods.  For example you could override the Write methods to periodically flush the stream.

Comments

Thank you for the answer. I will look into it.

Just for the record: This is what is written to the log using DownloadStream:

ERROR [FTPClient] 19 mai 2017 08:39:15.478 :  Caught exception Exception of type 'System.OutOfMemoryException' was thrown.
ERROR [FTPClient] 19 mai 2017 08:39:15.478 :  Caught exception
ERROR [FTPClient] 19 mai 2017 08:39:15.478 :  System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
ERROR [FTPClient] 19 mai 2017 08:39:15.478 :     at System.IO.MemoryStream.set_Capacity(Int32 value)
ERROR [FTPClient] 19 mai 2017 08:39:15.478 :     at System.IO.MemoryStream.EnsureCapacity(Int32 value)
ERROR [FTPClient] 19 mai 2017 08:39:15.478 :     at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
ERROR [FTPClient] 19 mai 2017 08:39:15.478 :     at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
ERROR [FTPClient] 19 mai 2017 08:39:15.478 :     at EnterpriseDT.Net.Ftp.FTPClient.ltRIXccKQk(Stream , String )
ERROR [FTPClient] 19 mai 2017 08:39:15.487 :  Exception in ValidateTransferOnError())
ERROR [FTPClient] 19 mai 2017 08:39:15.487 :  EnterpriseDT.Net.Ftp.FTPException: Connection closed; aborted transfer of "" (code=426)
ERROR [FTPClient] 19 mai 2017 08:39:15.487 :     at EnterpriseDT.Net.Ftp.FTPControlSocket.ValidateReply(FTPReply reply, String[] expectedReplyCodes)
ERROR [FTPClient] 19 mai 2017 08:39:15.487 :     at EnterpriseDT.Net.Ftp.FTPClient.ValidateTransfer()
ERROR [FTPClient] 19 mai 2017 08:39:15.487 :     at EnterpriseDT.Net.Ftp.FTPClient.ValidateTransferOnError()
ERROR [FTPClient] 19 mai 2017 08:39:15.487 :  SystemException in Get(Stream,string)
ERROR [FTPClient] 19 mai 2017 08:39:15.487 :  System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
ERROR [FTPClient] 19 mai 2017 08:39:15.487 :     at EnterpriseDT.Net.Ftp.FTPClient.ltRIXccKQk(Stream , String )
ERROR [FTPClient] 19 mai 2017 08:39:15.487 :     at EnterpriseDT.Net.Ftp.FTPClient.Get(Stream destStream, String remoteFile)

---

OK, this confirms that the exception is being thrown by the MemoryStream object that you passed in.

---

Agreed. So what I did was simple (ftpConn is the ExFTPConnection instance):
1. ftpConn.BytesTransferred += DownloadStreamBytesTransferred
2. ftpConn.TransferNotifyInterval = 32*1024
3. _zos.PutNextEntry(filePath)
4. ftpConn.DownloadStream(_zos,fileName)

And by letting DownloadStreamBytesTransferred() call _zos.Flush(), the OutOfMemory exception did not occur again.

Thanks a lot for the hint!

---

That should work fine.  BytesTransferred is called on the same thread that writes to the stream, so you shouldn't have any threading issues.