keepalive problems when using HTTP proxy

Question

We're trying to use an HTTP proxy with edtFTPj/PRO (3.1.0) and we're running into an issue.
The data channel is set up just fine and the contents is transferred, but the proxy doesn't
close down the data connection. That leaves the client hanging until it eventually times out
and reports a failure. I noticed on the connection setup that edtFTPj/PRO adds a keepalive
header, and I suspect that is what causes the proxy to behave the way it does.

Example: This is the data channel for a file listing command. First the HTTP request headers
from edtFTPj/PRO, then the HTTP response headers from Apache mod_proxy,
and at last the actual data (a directory listing with a single file).

CONNECT 69.xxx.xxx.xx:57043 HTTP/1.1
host: 69.xxx.xxx.xx:57043
user-agent: edtFTPj/PRO
proxy-connection: Keep-Alive
pragma: No-Cache

HTTP/1.0 200 Connection Established
Proxy-agent: Apache/2.3.6 (Unix) mod_ssl/2.3.6 OpenSSL/0.9.8g DAV/2

-rw-r--r-- 1 username groupname 242 Aug 13 14:56 dataset1.csv


At this point the connection should have been closed down by the proxy,
but it isn't, so edtFTPj/PRO waits for more data.

I found another forum topic which is similar: http://www.enterprisedt.com/forums/viewtopic.php?p=7262
However in our case we want to use the HTTP proxy.

We are using a prerelease of the HTTP proxy software <http://httpd.apache.org/docs/trunk/mod/mod_proxy.html>
so it could be that the proxy is to blame, it should know that there is no other way for the
client to detect when there is no more data on the data channel. On the other hand, the proxy
is trying to be very protocol agnostic when it comes to the data passing over a CONNECT
connection, so I'll cut it some slack here.

I don't see why we would want keepalive behavior in the first place. TCP level keepalive on the
control channel makes perfect sense, since there is nothing else going on there for the duration
of the file transfer. However, HTTP level keepalive on the data channel makes no sense to me.

I've been scouring the documentation to find if there is any way we can control the presence of
the keepalive HTTP header, but I can't find any. Is there any reason why the keepalive header
should be there at all? Is there anything I can do to prevent it?

Thanks in advance.

Answer 1

It's a good question. Here's a build without proxy-connection

http://www.enterprisedt.com/products/tm ... pj-pro.jar

Please try it out ...

Answer 2

Thanks a lot. We tried your build without the keepalive, and it still hits the timeout.
We think the reason is it is still an HTTP/1.1 call and the default is connection keepalive
in HTTP/1.1.

However, we tinkered with Apache, forcing its to timeout the client connection.
That is: http://httpd.apache.org/docs/2.2/mod/core.html#timeout was set to 10.
This made it work sort of ... after 10 seconds of course.

What we now think is that the CONNECT requests should specify 'Connection: close'
in the headers. Alternatively they could be HTTP/1.0. Does this make sense?

Answer 3

I've put up a build that uses 'Connection: close', which seems reasonable. Let us know how it goes.

Answer 4

Thanks again, I just tried with your latest jar. Unfortunately it doesn't change the behavior of our proxy.
We'll have to dig into the proxy in more detail to figure out what the culprit is. Thanks though. I'll let you
know what we find out.

Answer 5

It may be that HTTP 1.0 is required - it might assume 1.1 connections are persistent no matter what.