FTPHost.listdir fails for some servers if the path contains slashes

[ftputiluser]

Ftputil doesn't correctly list the content for some sites such as ftp.gnome.org .

>>> import ftputil
>>> conn=ftputil.FTPHost('ftp.gnome.org','anonymous','pippo@pippo.com')
>>> conn.listdir('/')
['pub']

(should be ['HEADER.html', 'Public', 'about', 'cdimage', 'conspiracy', 'debian', 'debian-cd', 'debian-non-US', 'favicon.ico', 'mirror', 'pub' , 'releases', 'robots.txt', 'ubuntu', 'welcome.msg', 'welcome2.msg']).

I think the problem is derived from python standard ftplib

>>> ftp = ftplib.FTP('ftp.gnome.org')
>>> ftp.login()
'230 Login successful.'

>>> ftp.dir('/')
drwxr-xr-x 4 ftp ftp 15 Dec 19 14:31 pub

However with the standard python library there is the following workaround:

>>> ftp.cwd(path)
>>> ftp.dir()

To apply this to ftputil it's enough to modify _robust_ftp_command method (line 562 in ftputil.py) and always use the code for 'special_case'.

If this small modification doesn't break anything I think it could be applied to ftputil,

regards Nicola

[ftputiluser]

Please note that the suggested workaround break the directory listing for ftp host suck as ftp.sunfreeware.com,

regards Nicola

[schwa]

Hi Nicola,

Thanks for reporting! :)

That server doesn't seem to like slashes in the path:

>>> conn.dir("")
-rw-r--r--    1 ftp      ftp          1001 May 07  2007 HEADER.html
lrwxrwxrwx    1 ftp      ftp             3 Sep 18  2008 Public -> pub
drwxr-xr-x    2 ftp      ftp             8 Apr 15 18:43 about
...
>>> conn.dir("/pub")
>>> conn.dir("pub")
lrwxrwxrwx    1 ftp      ftp            28 Sep 19  2008 EFLIB -> ../mirror/ftp.sunet.se/EFLIB
lrwxrwxrwx    1 ftp      ftp            19 Sep 19  2008 GNOME -> ../mirror/gnome.org
lrwxrwxrwx    1 ftp      ftp            14 Sep 19  2008 HEADER.html -> ../HEADER.html
...
>>> conn.dir("/mirror")
>>> conn.dir("mirror")
lrwxrwxrwx    1 ftp      ftp            14 Sep 18  2008 HEADER.html -> ../HEADER.html
drwxr-xr-x    3 ftp      ftp            25 Dec 23  2006 bittornado
drwxr-xr-x   11 ftp      ftp            13 May 09 09:00 cdimage.ubuntu.com
...
>>> conn.dir(".")
-rw-r--r--    1 ftp      ftp          1001 May 07  2007 HEADER.html
lrwxrwxrwx    1 ftp      ftp             3 Sep 18  2008 Public -> pub
drwxr-xr-x    2 ftp      ftp             8 Apr 15 18:43 about
...
>>> conn.dir("../../..")
drwxr-xr-x    4 ftp      ftp            15 Dec 19 14:31 pub
>>> conn.dir("../")
>>> conn.dir("../.")
>>>

You said the workaround to change to the directory first didn't work with ftp.sunfreeware.com. What did you try, what was the outcome, and what did you expect? Please show some code in the interpreter.

Stefan

[ftputiluser]

Hi Stefan,

sorry for the late response I see only now your response, here is the code:

standard ftputil special_case is not ever applied

In [1]: import ftputil

In [2]: conn=ftputil.FTPHost('ftp.gnome.org','anonymous','pippo@pippo.com')

In [3]: conn.listdir('/')
Out[3]: ['pub']

In [4]: conn=ftputil.FTPHost('ftp.sunfreeware.com','anonymous','pippo@pippo.com')

In [5]: conn.listdir('/')
Out[5]: ['bin', 'dev', 'etc', 'pub', 'usr']

patched ftputil special_case is always True:

import ftputil

In [2]: conn=ftputil.FTPHost('ftp.gnome.org','anonymous','pippo@pippo.com')

In [3]: conn.listdir('/')
Out[3]:                  
['HEADER.html',          
 'Public',               
 'about',                
 'cdimage',              
 'conspiracy',           
 'debian',               
 'debian-cd',            
 'debian-non-US',        
 'favicon.ico',          
 'mirror',               
 'pub',                  
 'releases',             
 'robots.txt',           
 'ubuntu',               
 'welcome.msg',          
 'welcome2.msg']         

In [4]: conn=ftputil.FTPHost('ftp.sunfreeware.com','anonymous','pippo@pippo.com')



In [5]: 

In [6]: 

In [7]: conn.listdir('/')
---------------------------------------------------------------------------
ParserError                               Traceback (most recent call last)

/home/nicola/workspace/FtpManager/<ipython console> 

/home/nicola/workspace/FtpManager/extlib/ftputil/ftputil.py in listdir(self, path)
    806         any of the available parsers raise a `ParserError`.               
    807         """                                                               
--> 808         return self._stat.listdir(path)                                   
    809                                                                           
    810     def lstat(self, path, _exception_for_missing_path=True):              

/home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in listdir(self, path)
    563         the server (e. g. timeout).                                         
    564         """                                                                 
--> 565         return self.__call_with_parser_retry(self._real_listdir, path)      
    566                                                                             
    567     def lstat(self, path, _exception_for_missing_path=True):                

/home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in __call_with_parser_retry(self, method, *args, **kwargs)                                                                                                    
    551                 self._allow_parser_switching = False                                                  
    552                 self._parser = MSParser()                                                             
--> 553                 return method(*args, **kwargs)
    554             else:
    555                 raise

/home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in _real_listdir(self, path)
    418             #  correct timestamp values in the cache
    419             stat_result = self._parser.parse_line(line,
--> 420                                                   self._host.time_shift())
    421             loop_path = self._path.join(path, stat_result._st_name)
    422             self._lstat_cache[loop_path] = stat_result

/home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in parse_line(self, line, time_shift)
    349                 st_size = int(dir_or_size)
    350             except ValueError:
--> 351                 raise ftp_error.ParserError("invalid size %s" % dir_or_size)
    352         else:
    353             st_size = None

ParserError: invalid size staff
Debugging info: ftputil 2.4.1, Python 2.4.3 (linux2)

Nicola

[schwa]

Listings as retrieved with ftp command line client

[schwa]

Hi Nicola,

thanks for following up. :-)

I tracked the problem down to this:

>>> import ftplib
>>> f = ftplib.FTP("ftp.sunfreeware.com", "anonymous", "sschwarzer@sschwarzer.net")
>>> f.dir("/")
total 10
lrwxrwxrwx   1 staff          7 Aug 13  2003 bin -> usr/bin
d--x--x--x   2 staff        512 Sep 24  2000 dev
d--x--x--x   3 staff        512 Sep 25  2000 etc
dr-xr-xr-x   3 staff        512 Oct  3  2000 pub
d--x--x--x   5 staff        512 Oct  3  2000 usr
>>> f.dir(".")
lrwxrwxrwx   1 staff          7 Aug 13  2003 bin -> usr/bin

dev:
total 10

etc:
total 10

pub:
total 4
-rw-r--r--   1 staff         74 Sep 25  2000 .message
----------   1 staff          0 Aug 16  2003 .notar
drwxr-xr-x  12 ftp          512 Nov 23  2008 freeware

usr:
total 4
>>> 

With the absolute path "/" as argument, the listing is retrieved as expected. With the relative path "." the server recurses down the current directory. I'll see if I can find a clean way to deal with this.

Stefan

[schwa]

Seen from a slightly more low-level perspective:

$ ftp -d ftp.sunfreeware.com
...
ftp> dir /
ftp: setsockopt (ignored): Permission denied
---> PORT 192,168,178,128,195,19
200 PORT command successful.
---> LIST /
150 Opening ASCII mode data connection for /bin/ls.
total 10
lrwxrwxrwx   1 staff          7 Aug 13  2003 bin -> usr/bin
d--x--x--x   2 staff        512 Sep 24  2000 dev
d--x--x--x   3 staff        512 Sep 25  2000 etc
dr-xr-xr-x   3 staff        512 Oct  3  2000 pub
d--x--x--x   5 staff        512 Oct  3  2000 usr
226 Transfer complete.

ftp> dir .
ftp: setsockopt (ignored): Permission denied
---> PORT 192,168,178,128,198,136
200 PORT command successful.
---> LIST .
150 Opening ASCII mode data connection for /bin/ls.
lrwxrwxrwx   1 staff          7 Aug 13  2003 bin -> usr/bin

dev: 
total 10

etc: 
total 10

pub: 
total 4
-rw-r--r--   1 staff         74 Sep 25  2000 .message
----------   1 staff          0 Aug 16  2003 .notar
drwxr-xr-x  12 ftp          512 Nov 23  2008 freeware

usr: 
total 4
226 Transfer complete.

ftp> dir
ftp: setsockopt (ignored): Permission denied
---> PORT 192,168,178,128,228,219
200 PORT command successful.
---> LIST
150 Opening ASCII mode data connection for /bin/ls.
total 10
lrwxrwxrwx   1 staff          7 Aug 13  2003 bin -> usr/bin
d--x--x--x   2 staff        512 Sep 24  2000 dev
d--x--x--x   3 staff        512 Sep 25  2000 etc
dr-xr-xr-x   3 staff        512 Oct  3  2000 pub
d--x--x--x   5 staff        512 Oct  3  2000 usr
226 Transfer complete.

Using no arguments after dir gives the same output as with "/" as argument whereas using "." as argument triggers the recursion.

[schwa]

Hi Nicola,

I committed a changeset 876 which should fix the problem. The former "special case" is now the default as you had originally suggested; FTPHost._robust_ftp_command is simplified a bit.

Please get the corresponding version of ftputil.py, test it and give feedback. Note: You only need ftputil.py; the other changes are only to test code. Thanks for your help! :o)

Stefan

[schwa]

Note: The patch should fix both problems, both ftp.gnome.org and ftp.sunfreeware.com should now behave as expected.

[ftputiluser]

Thanks Stefan,

the fix seems fine,

Nicola