FTPClient dir() method bug

Question

There appears to be a bug in the logic that a file parser, the UNIX variety anyway, follows.

Using the FTPClient method dir(<directoryname>, true) to obtain a full remote host directory listing against Solaris 9 or 10 results in the ftp server responding to a "LIST <directoryname>". The text returned is essentailly the same as would result from issuing an "ls -lR <directoryname>" at a shell prompt.

e.g. -> ls -lR .
.:
total 2946
drwxr-xr-x 2 oracle oinstall 512 Aug 1 10:19 C
drwxr-xr-x 2 oracle oinstall 4608 Jul 8 2005 PHSB1
-rwxr-xr-x 1 oracle oinstall 16988 Aug 1 10:19 cleanup
drwxr-xr-x 2 oracle oinstall 512 Jul 19 2005 ddl
-rw-r--r-- 1 oracle oinstall 20 Jun 24 2005 dual.sql
-rw-r--r-- 1 oracle oinstall 362 Jun 23 2005 loop10.lst
-rwxr-xr-x 1 oracle oinstall 574 Jun 23 2005 loop10.sh
-rwxr-xr-x 1 oracle oinstall 44 Jun 30 2005 mail.sh
-rw-r--r-- 1 oracle oinstall 44458 Jun 23 2005 phs1_counts.lst
-rw-r--r-- 1 oracle oinstall 44460 Jun 23 2005 phsb1_counts.lst
-rwxr-xr-x 1 oracle oinstall 149 Jun 23 2005 rc1.sh
-rw-r--r-- 1 oracle oinstall 912019 Nov 18 2005 scheduler_dc.dmp.Z
-rw-r--r-- 1 oracle oinstall 0 Jun 23 2005 separate.stderr
-rw-r--r-- 1 oracle oinstall 569 Jun 23 2005 separate.stdout
-rw-r--r-- 1 oracle oinstall 429029 Jun 23 2005 staging_schema.SQL
-rw-r--r-- 1 oracle oinstall 15860 Jun 23 2005 staging_schema.lst
-rw-r--r-- 1 oracle oinstall 198 Jun 23 2005 tablespaces_2.ddl
-rw-r--r-- 1 oracle oinstall 60 Jun 24 2005 test.sql
drwx------ 2 oracle oinstall 1024 Nov 8 2005 tmp

./C:
total 126
-rwxr-xr-x 1 oracle oinstall 16988 Aug 1 10:19 cleanup
-rw-r--r-- 1 oracle oinstall 6333 Aug 1 10:18 cleanup.c
-rwxr-xr-x 1 oracle oinstall 149 Aug 1 10:18 make_cleanup.sh
-rwxr-xr-x 1 oracle oinstall 10496 Jun 23 2005 separate
-rw-r--r-- 1 oracle oinstall 4536 Jun 23 2005 separate.c
-rw-r--r-- 1 oracle oinstall 0 Jun 23 2005 separate.stderr
-rw-r--r-- 1 oracle oinstall 1950 Jun 23 2005 separate.stdout
-rwxr-xr-x 1 oracle oinstall 10496 Nov 18 2005 trash
-rw-r--r-- 1 oracle oinstall 7573 Aug 1 10:18 utils.c
-rw-r--r-- 1 oracle oinstall 969 Aug 1 10:18 utils.h

./PHSB1:
total 0

./ddl:
total 240
-rw------- 1 oracle oinstall 348 Jun 30 2005 nohup.out
-rw-r--r-- 1 oracle oinstall 5594 Jul 19 2005 purch_dc_schema.lst
-rw-r--r-- 1 oracle oinstall 101072 Jul 19 2005 purch_dc_schema.sql
-rw-r--r-- 1 oracle oinstall 572 Jun 30 2005 tablespaces_2.ddl

./tmp:
total 131536
-rw-r--r-- 1 oracle oinstall 67293520 Nov 8 2005 nfsmaestroclient2006_evalesd.exe

Unfortunately when the FTPFileParser (an instance of UnixFileParser as one would expect) comes to process this, it chokes on lines like "./ddl:" with the following:

java.text.ParseException: Failed to parse line 'ddl:' (Unexpected number of fields in listing 'ddl:' - expected minimum 8 fields but found 1 fields)
at com.enterprisedt.net.ftp.FTPFileFactory.parse(FTPFileFactory.java:265)
at com.enterprisedt.net.ftp.FTPClient.dirDetails(FTPClient.java:2238)
etc. etc.

(Ignore the line number in FTPClient - I've modified the source to enable bandwidth throttling and on on the fly digest calculation so it's rather larger than the source as supplied).

You may also wish to note that when I say lines "like ./ddl:" it is just a likeness we are talking about. An "ls -lR" does produce the string "./ddl:" but the String returned from the ftp server responding to "LIST ." is actually "ddl:" although the line following is
"total 240" just like the listing above.

Fundamentally the problem seems to be that the UNIX parser doesn't know how to deal with recursive directory listings.
I know it's always a little tricky to determine the 'right' way of doing this but throwing a runtime exception is not it!

One question is obviously whether files in subdirectories should be included or just the directories themselves. If the files are included, in the context of an ftp client, should the name then be the full path name?

Personally I would opt for excluding subdirectory contents - just emulate the information returned from "ls -l" by a Unix ftp client and leave it at that.

At the time of writing I'm not sure what I will do to fix this issue. A quick hack that says "who cares if parsing fails" may suffice provided I suppress switching of parsers but my sense is that you need to look at this as an architectural issue and not just a problem in parsing recursive listings.

Cheers.

Answer 1

Hi

Thanks for you very detailed problem description. This is very surprising - so surprising in fact that I'm going to ask you to confirm that I've understood correctly. Are you saying that issuing a "LIST <directoryname>" command on solaris 9 and 10 results in a recursive directory listing!? I can't imagine why anyone would've chosen to implement the command like that - that's certainly not the intention of RFC959.

If that is really the case then we have to deal with it since we obviously want our software to work with Solaris. You are certainly right to say that it shouldn't throw an exception. I will discuss this with our parser-guru and try to come up with a solution.

- Hans (EDT)

Answer 2

Could you verify the type of FTP server (perhaps by the welcome message?). This is very unusual behaviour.

Also, can you FTP to it via a command-line client and type 'debug', followed by 'ls' or 'dir' - whichever one sends the LIST command - and email the output to support at enterprisedt dot com.

Answer 3

The ftp server is the standard Solaris in.ftpd daemon.

The Solaris 9 server is actually Solaris x86 a.k.a. Solaris for Intel:

[oracle@sunshine] /export/home/oracle
-> uname -a
SunOS sunshine 5.9 Generic_117172-17 i86pc i386 i86pc


The Solaris 10 server is a SunFire V440 SPARC box:

[oracle@delos] /export/home/oracle
-> uname -a
SunOS delos 5.10 Generic_118822-19 sun4u sparc SUNW,Sun-Fire-V440

I haven't seen any difference in their behaviour - Solaris tends to be good at that.

Using command line ftp the output from "ls -l" and "dir" are identical and the same as would be obtained from a shell prompt.

The output from "ls" is also the same as from a shell prompt.

Now, here's an interesting thing. Trying to re-create the problem or at least the diagnostics for same this morning I found that if I invoke dir(<directoryname>, true) WITHOUT first issuing a call to chdir(<directory>) there is no problem.

Furthermore, if I chdir(".") which is clearly useless but harmless there is still no problem.

Once a chdir(<some useful directory>) has been issued, the dir(<directoryname>, true) then behaves per the original 'bug' posting. (At an ftp comamnd line prompt the results is cd <directory> then "ls -l" are as expected).


My modified FTPClient.java code (from line 2105 in the original is):


LineNumberReader in = null;
Vector lines = new Vector();
try {
in = new LineNumberReader(
new InputStreamReader(data.getInputStream()));

// read a line at a time
String line = null;
while ((line = readLine(in)) != null)
{
lines.addElement(line);
log.info("FTP cmd " +command +" ->{" +line +"}"); // RSH debugging
}
}


The log.info is obviously the critical piece of information. Part of a sample run showing what happens is below:

>java -jar Z:\javaProjects\test\build\ftp.jar -hdelos -uoracle -p******
-d/u20/oradata/PWEB -D -tBIN -D PUT test.xml
INFO [com.enterprisedt.Demo] 17 Aug 2006 09:52:28.695 : Setting bandwidth to -1
INFO [com.enterprisedt.Demo] 17 Aug 2006 09:52:28.757 : Connecting
INFO [com.enterprisedt.Demo] 17 Aug 2006 09:52:28.961 : Logging in
INFO [com.enterprisedt.Demo] 17 Aug 2006 09:52:29.148 : Changing directory to /u
20/oradata/PWEB
INFO [com.enterprisedt.net.ftp.FTPClient] 17 Aug 2006 09:52:29.257 : Remote syst
em is UNIX Type: L8 Version: SUNOS
INFO [com.enterprisedt.net.ftp.FTPClient] 17 Aug 2006 09:52:29.507 : FTP cmd LIS
T . ->{-rw-r--r-- 1 oracle oinstall 9449 Aug 17 09:53 test.xml}
INFO [com.enterprisedt.net.ftp.FTPClient] 17 Aug 2006 09:52:29.507 : FTP cmd LIS
T . ->{-rw-r--r-- 1 oracle oinstall 8584 Feb 4 2006 weblimsphos.log}
INFO [com.enterprisedt.net.ftp.FTPClient] 17 Aug 2006 09:52:29.507 : FTP cmd LIS
T . ->{}
INFO [com.enterprisedt.net.ftp.FTPClient] 17 Aug 2006 09:52:29.507 : FTP cmd LIS
T . ->{arch:}
INFO [com.enterprisedt.net.ftp.FTPClient] 17 Aug 2006 09:52:29.507 : FTP cmd LIS
T . ->{total 2109312}
INFO [com.enterprisedt.net.ftp.FTPClient] 17 Aug 2006 09:52:29.507 : FTP cmd LIS
T . ->{-rw-r----- 1 oracle oinstall 19135695 Aug 11 09:04 log575046937_1_000
0004204.arc.Z}
INFO [com.enterprisedt.net.ftp.FTPClient] 17 Aug 2006 09:52:29.507 : FTP cmd LIS
T . ->{-rw-r----- 1 oracle oinstall 18905778 Aug 11 15:39 log575046937_1_000
0004205.arc.Z}

and so on.

An "ls -l /u20/oradata/PWEB" would show:

[oracle@delos] /u20/oradata/PWEB
-> ls -l
total 64
drwxr-xr-x 2 oracle oinstall 12288 Aug 17 10:11 arch
drwxr-xr-x 2 oracle oinstall 512 Jan 10 2006 ctl
-rw-r--r-- 1 oracle oinstall 9449 Aug 17 09:54 test.xml
-rw-r--r-- 1 oracle oinstall 8584 Feb 4 2006 weblimsphos.log

So recursion into the "arch" directory is taking place. (I omitted the full test output which shows recursion
into "ctl" as well).

As if that is not weird/annoying enough, I've experimented with the code that invokes dirDetails() in the calling program
and found more quirks.

Normally when using ftp at the command line, if one wants to get a full directory listing, the order of events would be

1) ftp <hostname>
2) <supply credentials>
3) cd <directory>
4) ls -l

Using
4) ls -l <directory>
jus doesn't work with or without step 3) - although ls <directory> does irrespective of execution of step 3).

Anyhow, in coding the calls to FTPClient, I just followed the steps 1) to 4) approach above and consequently issued
a dirDetails("."). If, instead, I issue a dirDetails(<directory I have changed to>), everything works.

And I haven't even begun to visit the fairy tale (witches?) world of Microsoft!

CONCLUSION
==========

IT'S NOT WORTH LOSING ANY SLEEP OVER THIS - WE/I JUST NEED TO CODE 'CORRECTLY'

Cheers.