SFTP and enabling compression

Question

Hello,

I am writing a SFTP client connecting to a GLOBALscape FTP server and am trying to enable compression when sending and getting files.

I have implemented the connection as follows:
ftp = new SSHFTPClient();

ProxySettings ps = ftp.getProxySettings();
// Socks configuration affects all TCP connections in this VM
if (configuration.getSocksProxyHost() != null)
{
ps.setProxyType(ProxyType.SOCKS5);
ps.setProxyAddress(configuration.getSocksProxyHost());
ps.setProxyPort(Integer.parseInt(configuration.getSocksProxyPort()));
}
if (configuration.getSocksProxyUsername() != null)
{
ps.setProxyUserName(configuration.getSocksProxyUsername());
ps.setProxyPassword(configuration.getSocksProxyPassword());
}
// The host public key file should be relative to the config directory.
File hostKeyFile = new File(configuration.getKnownHostsFile());

if (!hostKeyFile.exists())
{
//throw new Exception("host key file doesn't exist. " + hostKeyFile.getAbsolutePath());
}

ftp.setRemotePort(configuration.getFTPSPort());
ftp.setRemoteHost(configuration.getRemoteHost());
ftp.getValidator().loadKnownHosts(hostKeyFile.getAbsolutePath());
ftp.setAuthentication(configuration.getUsername(),configuration.getPassword());

ftp.disableAllAlgorithms(SSHFTPAlgorithm.COMPRESSION);
ftp.setAlgorithmEnabled(SSHFTPAlgorithm.COMPRESSION_ZLIB, true);
ftp.setMaxPacketSize(64*1024);
ftp.setParallelMode(true);
ftp.setTransferBufferSize(64*1024);

ftp.connect();


When I use this was to connect, I usually get compression when putting a file to the FTP server but I never seem to get compression when getting a file from the server.

Any ideas on what I am doing wrong?

Thanks.

Answer 1

Please post (or email to support at enterprisedt dot com) a log file at the debug level. That'll tell us what's happening.

FYI once compression is selected, it applies to everything in that session - so you can't have upload being compressed and download not being compressed in the same session.

Answer 2

Hello,

The log file is very large. I will post up to the first 1000 lines of the log.

After more investigation, it does appear that the compression is happening both sending and receiving files. What actually is happening is that the
connection speed seems to throttle down for files larger than 100 mb. The transfer starts off quick (I can see this by monitoring my GLOBALscape
FTP server and then after about 10 seconds the transfer speed drops down to a crawl. I have tried WinSCP as a client to connect to GLOBALscape
using SFTP and it does not exhibit this issue with getting files. Do you have any insight as to why this is happening?

Thanks


INFO 2010-11-11 08:35:35,908 RapidResponse.DataTransferClient.Configuration.Change - New configuration was loaded successfully.
INFO 2010-11-11 08:35:35,958 RapidResponse.DataTransferClient.Monitor.Start - Starting to monitor Extract directories.
DEBUG 2010-11-11 08:35:36,140 CryptixProperties - Successfully loaded the CryptixEDT properties file
DEBUG 2010-11-11 08:35:36,140 CryptixProperties - Listing CryptixEDT properties
DEBUG 2010-11-11 08:35:36,140 CryptixProperties - Alg.Alias.PaddingScheme.PKCS5Padding=PKCS#5
DEBUG 2010-11-11 08:35:36,140 CryptixProperties - Alg.Alias.Cipher.AES=Rijndael
DEBUG 2010-11-11 08:35:36,141 CryptixProperties - Msg.ok="OK"
DEBUG 2010-11-11 08:35:36,141 CryptixProperties - Msg.commandline.cancel="c"
DEBUG 2010-11-11 08:35:36,141 CryptixProperties - Mode.PCBC=com.enterprisedt.cryptix.provider.mode.PCBC
DEBUG 2010-11-11 08:35:36,141 CryptixProperties - Alg.Alias.Cipher.TripleDES=DES-EDE3
DEBUG 2010-11-11 08:35:36,141 CryptixProperties - Alg.bitLength.HAVAL=256
DEBUG 2010-11-11 08:35:36,141 CryptixProperties - KeyGenerator.DES=com.enterprisedt.cryptix.provider.key.DESKeyGenerator
DEBUG 2010-11-11 08:35:36,141 CryptixProperties - KeyGenerator.RPK=com.enterprisedt.cryptix.provider.rpk.RPKKeyGenerator
DEBUG 2010-11-11 08:35:36,141 CryptixProperties - MessageDigest.HMAC-RIPEMD128=com.enterprisedt.cryptix.provider.mac.HMAC_RIPEMD128
DEBUG 2010-11-11 08:35:36,141 CryptixProperties - Alg.Alias.Signature.SHA-1/ElGamal=SHA-1/ElGamal/PKCS#1
DEBUG 2010-11-11 08:35:36,141 CryptixProperties - Msg.no="No"
DEBUG 2010-11-11 08:35:36,142 CryptixProperties - Alg.Alias.Signature.SHA-1/RPK=SHA-1/RPK/PKCS#1
DEBUG 2010-11-11 08:35:36,142 CryptixProperties - Debug.Level.RIPEMD160=0
DEBUG 2010-11-11 08:35:36,142 CryptixProperties - Msg.warning="Warning: "
DEBUG 2010-11-11 08:35:36,142 CryptixProperties - MessageDigest.HMAC-HAVAL=com.enterprisedt.cryptix.provider.mac.HMAC_HAVAL
DEBUG 2010-11-11 08:35:36,142 CryptixProperties - Alg.Alias.MessageDigest.{1=3 36 3 2 1} = RIPEMD160
DEBUG 2010-11-11 08:35:36,142 CryptixProperties - Alg.OID.Cipher.RC2/CBC/PKCS#5={1 2 840 113549 3 2}
DEBUG 2010-11-11 08:35:36,142 CryptixProperties - Alg.Alias.Signature.SHA/DSA=SHA1withDSA
DEBUG 2010-11-11 08:35:36,142 CryptixProperties - Debug.Level.SPEED=0
DEBUG 2010-11-11 08:35:36,142 CryptixProperties - Signature.RawRSA=com.enterprisedt.cryptix.provider.rsa.RSASignature
DEBUG 2010-11-11 08:35:36,142 CryptixProperties - Alg.Alias.KeyGenerator.AES=Rijndael
DEBUG 2010-11-11 08:35:36,142 CryptixProperties - Alg.Alias.Cipher.LOKI=LOKI91
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Alg.rounds.SPEED=64
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Msg.error="Error: "
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Msg.cancel="Cancel"
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Alg.Alias.KeyPairGenerator.DH=ElGamal
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Cipher.DESX=com.enterprisedt.cryptix.provider.cipher.DESX
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Cipher.AES=com.enterprisedt.cryptix.provider.cipher.Rijndael
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - KeyPairGenerator.RSA=com.enterprisedt.cryptix.provider.rsa.BaseRSAKeyPairGenerator
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - KeyGenerator.RC4=com.enterprisedt.cryptix.provider.key.RC4KeyGenerator
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - KeyGenerator.RC2=com.enterprisedt.cryptix.provider.key.RC2KeyGenerator
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Msg.commandline.ok_cancel=" : (ok/cancel) "
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - PGP.ConfigPath=~/.pgp/config.txt
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Alg.Alias.PaddingScheme.PKCS7=PKCS#7
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Alg.OID.MessageDigest.SHA-1={1 3 14 3 2 26}
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Alg.Alias.PaddingScheme.PKCS5=PKCS#5
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Debug.Level.LOKI91=0
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Debug.Level.IDEA=0
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Cipher.Square=com.enterprisedt.cryptix.provider.cipher.Square
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Alg.rounds.Blowfish=16
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - KeyGenerator.CAST5=com.enterprisedt.cryptix.provider.key.CAST5KeyGenerator
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Debug.Level.Blowfish=0
DEBUG 2010-11-11 08:35:36,143 CryptixProperties - Alg.Alias.KeyGenerator.CAST=CAST5
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Debug.Level.MD5=0
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Signature.MD5/ElGamal/PKCS#1=com.enterprisedt.cryptix.provider.elgamal.MD5_ElGamal_PKCS1Signature
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Signature.RawDSA=com.enterprisedt.cryptix.provider.dsa.RawDSASignature
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Alg.Alias.MessageDigest.RIPEMD-160=RIPEMD160
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Alg.Alias.KeyGenerator.LOKI-91=LOKI91
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Alg.OID.Cipher.RSA/PKCS#1={1 2 840 113549 1 1 1}
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Alg.Alias.MessageDigest.HMAC-RIPEMD-160=HMAC-RIPEMD160
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - PaddingScheme.PKCS#7=com.enterprisedt.cryptix.provider.padding.PKCS7
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Alg.Alias.Signature.MD2/RSA=MD2/RSA/PKCS#1
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - KeyGenerator.SPEED=com.enterprisedt.cryptix.provider.key.SPEEDKeyGenerator
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - PaddingScheme.PKCS#5=com.enterprisedt.cryptix.provider.padding.PKCS5
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Alg.Alias.KeyGenerator.CAST-128=CAST5
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Cipher.ARC4=com.enterprisedt.cryptix.provider.cipher.RC4
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Alg.Alias.KeyGenerator.DES-2X=DES2X
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Mode.CBC=com.enterprisedt.cryptix.provider.mode.CBC
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - MessageDigest.RIPEMD160=com.enterprisedt.cryptix.provider.md.RIPEMD160
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Msg.commandline.yes_no_cancel=" : (yes/no/cancel) "
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - MessageDigest.SHA-1=com.enterprisedt.cryptix.provider.md.SHA1
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - MessageDigest.SHA-0=com.enterprisedt.cryptix.provider.md.SHA0
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Alg.OID.Cipher.RC5/CBC/PKCS#5={1 2 840 113549 3 9}
DEBUG 2010-11-11 08:35:36,144 CryptixProperties - Debug.Level.SAFER=0
DEBUG 2010-11-11 08:35:

Answer 3

It's probably best if you email us the log file. If you zip it up it shouldn't be too large hopefully.