Our Products:   CompleteFTP  edtFTPnet/Free  edtFTPnet/PRO  edtFTPj/Free  edtFTPj/PRO
0 votes
4.9k views
in .NET FTP by (260 points)
We are creating a site application which will be reading and parsing log files from various other servers. These logs will be several hundred MB each so we don't want to download the logs to be read. Is it possible to resume a downloadstream if you supply the byte count as a starting point?

If you can, how would you code it.

Thanks

6 Answers

0 votes
by (162k points)
If you are using FTP yes, it is possible to use the REST command to retrieve from any given point in the remote file.

PS if you are downloading log files, use a server that supports MODE Z compression to speed up your transfers hugely. edtFTPnet/Express and edtFTPnet/PRO support MODE Z compression.
0 votes
by (260 points)
never used ftp before but this seems to work...

        Dim LogDL As FTPConnection = New FTPConnection

        'setup ftp
        LogDL.AutoLogin = True
        LogDL.ConnectMode = FTPConnectMode.ACTIVE
        LogDL.ParsingCulture = New System.Globalization.CultureInfo("")
        LogDL.UserName = ftpUser
        LogDL.Password = ftpPassword
        LogDL.ServerAddress = ftpHost
        LogDL.ServerPort = ftpPort
        LogDL.StrictReturnCodes = True
        LogDL.Timeout = 5000
        LogDL.TransferBufferSize = 4096
        LogDL.TransferNotifyInterval = CType(4096, Long)
        LogDL.TransferType = FTPTransferType.BINARY
 
       Try
            'connect
            LogDL.Connect()
            'change remote directory
            LogDL.ChangeWorkingDirectory(ftpRemDir)
            'Get The file
            LogDL.CloseStreamsAfterTransfer = False
            Dim str As MemoryStream = New MemoryStream()
            LogDL.DownloadStream(str, RemFName)
            str.Seek(LastByte, SeekOrigin.Begin)
            Dim sr As New StreamReader(str)
            'remove next line
            Dim sw As New StreamWriter(ftpLocalDir + "streamed.log")
            Dim itm As Object
            'reading line by line 
            itm = sr.ReadLine
            Do While sr.Peek() >= 0
                'discard empty lines
                If itm > Nothing Then
                    'replace with temp db insert function
                    sw.WriteLine(itm)
                End If
                'read next line from stream 
                itm = sr.ReadLine
            Loop
            'catch bytecount to be inserted to db
            Dim ByteCnt As Integer = str.Length
            txtResponse.Text = ByteCnt
            sr.Close()
            sw.Close()
            'Close the connection
            LogDL.Close()

        Catch ex As Exception
            txtResponse.Text = vbCrLf & ex.Message
        End Try


Is there a better way or is this sufficient.

and how would i implement mode z?
0 votes
by (162k points)
Just FYI you are downloading the entire file here.

MODE Z is supported in edtFTPnet/Express and edtFTPnet/PRO. You set the property "CompressionPreferred' to true. Of course the server needs to support MODE Z for compression to work.

But with log files you will achieve probably 95% compression (if they aren't already zipped of course) - a 200 MB file will probably compress to perhaps 10 MB.
0 votes
by (260 points)
can't find the compression option (i'm testing with the free version).

Can you give me an example in vb for using the rest command as this seems to be my only option with this version.

Thanks
0 votes
by (260 points)
it doesn't matter (altho it would still be good to know) because I've checked the ftp server logs and using the above code the logfile is resumed from the position given in the seek command.
0 votes
by (162k points)
Ok good that it resumes correctly. Compression isn't supported in the free version, you need the express or pro version.

Categories

...