edtFTPnet/Free cutting files issue

Question

We have been using edtFTPnet/Free for many years without any issues (since 2009). Recently we've moved our service (based on edtFTPnet/Free) to modern OS (Windows 2016). Now we are experiencing strange behaviour - sometimes (once or twice a day) file that is uploaded to FTP is incomplete/corrupted.

I am not sure if it is related to the edt library but it is worth asking.

Algorithm is very simple - our service uploads a file (for each file in given directory) as a temporary file, and then renames it to original name.

Since it's kind of random issue (most files are uploaded ok) - is it some kind of compatibility issue or some kind of known problem in Windows 2016. If that's the case - how may I try to solve it (or maybe it's not an issue with edt at all).

Regards

 Tomek

Comments

I would enable debug logging and examine the logs to see if they give you any clues as to why this is happening. It's possible there are intermittent connectivity issues on the new machine for example.

---

1. I've tried debugging - but nothing happens during debug session.
2. I've added file sizes (local and remote) - it shows me that sometimes local file size is bigger than remote (btw. is there an easy way to read remote file size - my way is to use GetFileInfos() method and get size property for given file name).
3. I've found some interesting ideas that it may be related to TCP Offloading (https://serverfault.com/questions/574460/ftp-partial-uploads-on-iis8, https://enterprisedt.com/forums/viewtopic.php?f=5&t=4542), so I've disabled: IPv4 Checksum Offload, IPv4 TSO Offload, Large Send Offload v2 (IPv4), Large Send Offload v2 (IPv6), TCP Checksum Offload (IPv4), TCP Checksum Offload (IPv6). Since then one file became truncated. Maybe there are some other options that I am not aware of, that should be turned off (I've left enabled Offload IP Options, Offload Tagged Traffic, Offload TCP Options, UDP Checksum Offload (IPv4), UDP Checksum Offload (IPv6) ).
4. I don't think it is related to internet connectivity, becouse I am uploading file as TMP, and then rename to original file name. In case of internet failure there should be TMP file on FTP (instead of the original one).

Regards]

  Tomek

---

I've disabled Offload IP Options, Offload Tagged Traffic, Offload TCP Options, UDP Checksum Offload (IPv4), UDP Checksum Offload (IPv6), Recv Segment Coalescing (IPV4) and Recv Segment Coalescing (IPV6. Still no luck.

So I've enabled edtFTPNet debug logging (LogLevel=All). It looks like everything went ok, but file is truncated (bytes transferred is greater than file size stored on FTP):

DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.049 : 257 "/xxx/lprb/inbox" is the current directory
DEBUG [FTPConnection] 5 maj 2020 10:11:52.049 : UploadFile(C:\EDI_MZ\FTPConnect.data\xxx\outbox\lprb\LPSF20_08368.txt,LPSF20_08368.txt.tmp)
DEBUG [FTPConnection] 5 maj 2020 10:11:52.049 : Combining absolute path 'C:\Windows\system32' with relative path 'C:\EDI_MZ\FTPConnect.data\xxx\outbox\lprb\LPSF20_08368.txt'
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.049 : ---> PASV
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.065 : 227 Entering Passive Mode (xx,xx,xx,xx,196,178).
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.065 : Server supplied address=xx.xx.xx.xx
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.065 : Server supplied port=50354
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.065 : Substituting server supplied IP (xx.xx.xx.xx) with remote host IP (xx.xx.xx.xx)
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.065 : NewPassiveDataSocket(xx.xx.xx.xx,50354)
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.065 : ---> STOR LPSF20_08368.txt.tmp
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.065 : 150 Opening BINARY mode data connection for LPSF20_08368.txt.tmp
DEBUG [FTPClient] 5 maj 2020 10:11:52.065 : Closing source stream
DEBUG [FTPClient] 5 maj 2020 10:11:52.065 : Transferred 5348 bytes to remote host
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.081 : 226 Transfer complete
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.081 : ---> RNFR LPSF20_08368.txt.tmp
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.081 : 350 File or directory exists, ready for destination name
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.081 : ---> RNTO LPSF20_08368.txt
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.081 : 250 Rename successful
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.081 : ---> SYST
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.096 : 215 UNIX Type: L8
DEBUG [FTPFileFactory] 5 maj 2020 10:11:52.096 : Selected UNIX parser
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.096 : ---> PWD
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.096 : 257 "/xxx/lprb/inbox" is the current directory
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.096 : ---> PASV
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.096 : 227 Entering Passive Mode (xx,xx,xx,xx,198,110).
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.112 : Server supplied address=xx.xx.xx.xx
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.112 : Server supplied port=50798
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.112 : Substituting server supplied IP (xx.xx.xx.xx) with remote host IP (xx.xx.xx.xx)
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.112 : NewPassiveDataSocket(xx.xx.xx.xx,50798)
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.112 : ---> LIST
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.112 : 150 Opening BINARY mode data connection for file list
DEBUG [FTPClient] 5 maj 2020 10:11:52.112 : Reading ASCII listing data
DEBUG [FTPClient] 5 maj 2020 10:11:52.112 : -->drwxr-xr-x   2 4174     65532        4096 May  5 10:11 .
DEBUG [FTPClient] 5 maj 2020 10:11:52.112 : -->drwxr-xr-x   4 4174     65532        4096 Jan  5  2016 ..
DEBUG [FTPClient] 5 maj 2020 10:11:52.112 : -->-rw-r--r--   1 4174     65532        4096 May  5 10:11 LPSF20_08368.txt
DEBUG [FTPControlSocket] 5 maj 2020 10:11:52.128 : 226 Transfer complete

 

 

---

Have you downloaded the transferred file using Filezilla or some other client and compared them to see what the difference is? Does uploading with a different client produce the same result?

---

It's hard to tell, since it's kind of a random issue. During uploading with different client everything seems to be working ok. On the other hand most files uploaded with edtFTP is also ok. During debug session everything is also working ok. Even this piece of software (my service based on edtFTP) works for most of the time (and it has been working ok for about 10 years). For example since Thursday untill Monday every file was uploaded ok, but today 2 uploaded files are truncated.
The only change that I am aware of was moving my service (based on edtFTP) form Windows 2003 to Windows 2016 (FTP server remains the same) - but I don't know how could it matters.
One more thing - every truncated file has size x*4096 bytes - maybe it's a coincidence, or maybe a hint.

---

If I were you, I'd put my upload in a loop where it keeps resuming the upload until the size of the uploaded file matches the size of the source file. Of course, this won't work with ASCII files since the size of the file can change as EOLs are converted, but if your transfers are binary then it should work.

Also, many servers implement the XCRC command, which will allow you to check for corruption of the file, other than simple truncation, so you can include that check.

These are good practices in general anyway, so I'd definitely recommend that you implement them.

I just remembered that our commercial product, edtFTPnet/PRO, has an 'integrity checking' feature that you can enable to have CRC checks automatically performed after each transfer.

---

I was thinking about similar workaround (if there is a size mismatch then delete file and upload it once again).
1. XCRC/integrity checking is available only in PRO version?
2. What about FTP file size checking - is there an easy way of checking file that has been uploaded? Or my solution is ok (I don't like it): upload file, then check size for every file in FTP directory and find the one with the correct name.

      static long remoteFileSize(string _fileName, FTPConnection _ftpCon)
        {
            FTPFile[] fileDetails = _ftpCon.GetFileInfos();
            long remotesize = 0 ;

            foreach (FTPFile ftpfiles in fileDetails)
            {
                if (ftpfiles.Name == _fileName)
                {
                    remotesize = ftpfiles.Size;
                    
                }
            }
            return remotesize;
        }

BTW. In one of my previous comments I was able to change fonts, mark quotation, etc. In this post I can only write in simple text (none of the formatting options is available). Why?

---

Yes, integrity checking is only in the Pro version, but you should be able to implement it yourself without too much trouble.

I'd use the GetSize method rather than GetFileInfos as it's more efficient.

About formatting, yes, it's pretty annoying. It's only available for answers, not comments. I've got no idea why the developers of Q2A chose to do that, but then it's open source, so we can't really complain.

---

I was unable to find exact couse, but your suggestion was very good. I've implemented checking size (local and remote). If remote is different than local, the remote file is deleted (and reuploaded in next iteration). It seems to be working ok, and the issue is somehow resolved (errors still appear, but due to this workaround it's no longer a big issue).
Thanks everyone for your help.

Answer 1

Tomasz found a solution, which he described in the comment (above):

I've implemented checking size (local and remote). If remote is different than local, the remote file is deleted (and reuploaded in next iteration). It seems to be working ok, and the issue is somehow resolved (errors still appear, but due to this workaround it's no longer a big issue).