Opened 10 years ago

Closed 9 years ago

Last modified 9 years ago

#33 closed defect (fixed)

FTPHost.listdir fails for some servers if the path contains slashes

Reported by: ftputiluser Owned by: schwa
Priority: major Milestone: 2.4.2
Component: Library Version: 2.4
Keywords: listdir, change directory, special_case Cc:

Description (last modified by schwa)

Ftputil doesn't correctly list the content for some sites such as ftp.gnome.org .

>>> import ftputil
>>> conn=ftputil.FTPHost('ftp.gnome.org','anonymous','pippo@pippo.com')
>>> conn.listdir('/')
['pub']

(should be ['HEADER.html', 'Public', 'about', 'cdimage', 'conspiracy', 'debian', 'debian-cd', 'debian-non-US', 'favicon.ico', 'mirror', 'pub' , 'releases', 'robots.txt', 'ubuntu', 'welcome.msg', 'welcome2.msg']).

I think the problem is derived from python standard ftplib

>>> ftp = ftplib.FTP('ftp.gnome.org')
>>> ftp.login()
'230 Login successful.'

>>> ftp.dir('/')
drwxr-xr-x 4 ftp ftp 15 Dec 19 14:31 pub

However with the standard python library there is the following workaround:

>>> ftp.cwd(path)
>>> ftp.dir()

To apply this to ftputil it's enough to modify _robust_ftp_command method (line 562 in ftputil.py) and always use the code for 'special_case'.

If this small modification doesn't break anything I think it could be applied to ftputil,

regards Nicola

Attachments (1)

listings.txt (1.7 KB) - added by schwa 9 years ago.
Listings as retrieved with ftp command line client

Download all attachments as: .zip

Change History (11)

comment:1 Changed 10 years ago by schwa

  • Description modified (diff)
  • Keywords listdir change directory special_case added
  • Status changed from new to assigned
  • Version set to 2.4

comment:2 Changed 10 years ago by ftputiluser

Please note that the suggested workaround break the directory listing for ftp host suck as ftp.sunfreeware.com,

regards Nicola

comment:3 Changed 10 years ago by schwa

  • Summary changed from Ftputil listdir broken with some site to FTPHost.listdir fails for some servers if the path contains slashes

Hi Nicola,

Thanks for reporting! :)

That server doesn't seem to like slashes in the path:

>>> conn.dir("")
-rw-r--r--    1 ftp      ftp          1001 May 07  2007 HEADER.html
lrwxrwxrwx    1 ftp      ftp             3 Sep 18  2008 Public -> pub
drwxr-xr-x    2 ftp      ftp             8 Apr 15 18:43 about
...
>>> conn.dir("/pub")
>>> conn.dir("pub")
lrwxrwxrwx    1 ftp      ftp            28 Sep 19  2008 EFLIB -> ../mirror/ftp.sunet.se/EFLIB
lrwxrwxrwx    1 ftp      ftp            19 Sep 19  2008 GNOME -> ../mirror/gnome.org
lrwxrwxrwx    1 ftp      ftp            14 Sep 19  2008 HEADER.html -> ../HEADER.html
...
>>> conn.dir("/mirror")
>>> conn.dir("mirror")
lrwxrwxrwx    1 ftp      ftp            14 Sep 18  2008 HEADER.html -> ../HEADER.html
drwxr-xr-x    3 ftp      ftp            25 Dec 23  2006 bittornado
drwxr-xr-x   11 ftp      ftp            13 May 09 09:00 cdimage.ubuntu.com
...
>>> conn.dir(".")
-rw-r--r--    1 ftp      ftp          1001 May 07  2007 HEADER.html
lrwxrwxrwx    1 ftp      ftp             3 Sep 18  2008 Public -> pub
drwxr-xr-x    2 ftp      ftp             8 Apr 15 18:43 about
...
>>> conn.dir("../../..")
drwxr-xr-x    4 ftp      ftp            15 Dec 19 14:31 pub
>>> conn.dir("../")
>>> conn.dir("../.")
>>>

You said the workaround to change to the directory first didn't work with ftp.sunfreeware.com. What did you try, what was the outcome, and what did you expect? Please show some code in the interpreter.

Stefan

comment:4 Changed 9 years ago by ftputiluser

Hi Stefan,

sorry for the late response I see only now your response, here is the code:

standard ftputil special_case is not ever applied

In [1]: import ftputil

In [2]: conn=ftputil.FTPHost('ftp.gnome.org','anonymous','pippo@pippo.com')

In [3]: conn.listdir('/')
Out[3]: ['pub']

In [4]: conn=ftputil.FTPHost('ftp.sunfreeware.com','anonymous','pippo@pippo.com')

In [5]: conn.listdir('/')
Out[5]: ['bin', 'dev', 'etc', 'pub', 'usr']

patched ftputil special_case is always True:

import ftputil

In [2]: conn=ftputil.FTPHost('ftp.gnome.org','anonymous','pippo@pippo.com')

In [3]: conn.listdir('/')
Out[3]:                  
['HEADER.html',          
 'Public',               
 'about',                
 'cdimage',              
 'conspiracy',           
 'debian',               
 'debian-cd',            
 'debian-non-US',        
 'favicon.ico',          
 'mirror',               
 'pub',                  
 'releases',             
 'robots.txt',           
 'ubuntu',               
 'welcome.msg',          
 'welcome2.msg']         

In [4]: conn=ftputil.FTPHost('ftp.sunfreeware.com','anonymous','pippo@pippo.com')



In [5]: 

In [6]: 

In [7]: conn.listdir('/')
---------------------------------------------------------------------------
ParserError                               Traceback (most recent call last)

/home/nicola/workspace/FtpManager/<ipython console> 

/home/nicola/workspace/FtpManager/extlib/ftputil/ftputil.py in listdir(self, path)
    806         any of the available parsers raise a `ParserError`.               
    807         """                                                               
--> 808         return self._stat.listdir(path)                                   
    809                                                                           
    810     def lstat(self, path, _exception_for_missing_path=True):              

/home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in listdir(self, path)
    563         the server (e. g. timeout).                                         
    564         """                                                                 
--> 565         return self.__call_with_parser_retry(self._real_listdir, path)      
    566                                                                             
    567     def lstat(self, path, _exception_for_missing_path=True):                

/home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in __call_with_parser_retry(self, method, *args, **kwargs)                                                                                                    
    551                 self._allow_parser_switching = False                                                  
    552                 self._parser = MSParser()                                                             
--> 553                 return method(*args, **kwargs)
    554             else:
    555                 raise

/home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in _real_listdir(self, path)
    418             #  correct timestamp values in the cache
    419             stat_result = self._parser.parse_line(line,
--> 420                                                   self._host.time_shift())
    421             loop_path = self._path.join(path, stat_result._st_name)
    422             self._lstat_cache[loop_path] = stat_result

/home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in parse_line(self, line, time_shift)
    349                 st_size = int(dir_or_size)
    350             except ValueError:
--> 351                 raise ftp_error.ParserError("invalid size %s" % dir_or_size)
    352         else:
    353             st_size = None

ParserError: invalid size staff
Debugging info: ftputil 2.4.1, Python 2.4.3 (linux2)

Nicola

Changed 9 years ago by schwa

Listings as retrieved with ftp command line client

comment:5 Changed 9 years ago by schwa

Hi Nicola,

thanks for following up. :-)

I tracked the problem down to this:

>>> import ftplib
>>> f = ftplib.FTP("ftp.sunfreeware.com", "anonymous", "sschwarzer@sschwarzer.net")
>>> f.dir("/")
total 10
lrwxrwxrwx   1 staff          7 Aug 13  2003 bin -> usr/bin
d--x--x--x   2 staff        512 Sep 24  2000 dev
d--x--x--x   3 staff        512 Sep 25  2000 etc
dr-xr-xr-x   3 staff        512 Oct  3  2000 pub
d--x--x--x   5 staff        512 Oct  3  2000 usr
>>> f.dir(".")
lrwxrwxrwx   1 staff          7 Aug 13  2003 bin -> usr/bin

dev:
total 10

etc:
total 10

pub:
total 4
-rw-r--r--   1 staff         74 Sep 25  2000 .message
----------   1 staff          0 Aug 16  2003 .notar
drwxr-xr-x  12 ftp          512 Nov 23  2008 freeware

usr:
total 4
>>> 

With the absolute path "/" as argument, the listing is retrieved as expected. With the relative path "." the server recurses down the current directory. I'll see if I can find a clean way to deal with this.

Stefan

comment:6 Changed 9 years ago by schwa

Seen from a slightly more low-level perspective:

$ ftp -d ftp.sunfreeware.com
...
ftp> dir /
ftp: setsockopt (ignored): Permission denied
---> PORT 192,168,178,128,195,19
200 PORT command successful.
---> LIST /
150 Opening ASCII mode data connection for /bin/ls.
total 10
lrwxrwxrwx   1 staff          7 Aug 13  2003 bin -> usr/bin
d--x--x--x   2 staff        512 Sep 24  2000 dev
d--x--x--x   3 staff        512 Sep 25  2000 etc
dr-xr-xr-x   3 staff        512 Oct  3  2000 pub
d--x--x--x   5 staff        512 Oct  3  2000 usr
226 Transfer complete.

ftp> dir .
ftp: setsockopt (ignored): Permission denied
---> PORT 192,168,178,128,198,136
200 PORT command successful.
---> LIST .
150 Opening ASCII mode data connection for /bin/ls.
lrwxrwxrwx   1 staff          7 Aug 13  2003 bin -> usr/bin

dev: 
total 10

etc: 
total 10

pub: 
total 4
-rw-r--r--   1 staff         74 Sep 25  2000 .message
----------   1 staff          0 Aug 16  2003 .notar
drwxr-xr-x  12 ftp          512 Nov 23  2008 freeware

usr: 
total 4
226 Transfer complete.

ftp> dir
ftp: setsockopt (ignored): Permission denied
---> PORT 192,168,178,128,228,219
200 PORT command successful.
---> LIST
150 Opening ASCII mode data connection for /bin/ls.
total 10
lrwxrwxrwx   1 staff          7 Aug 13  2003 bin -> usr/bin
d--x--x--x   2 staff        512 Sep 24  2000 dev
d--x--x--x   3 staff        512 Sep 25  2000 etc
dr-xr-xr-x   3 staff        512 Oct  3  2000 pub
d--x--x--x   5 staff        512 Oct  3  2000 usr
226 Transfer complete.

Using no arguments after dir gives the same output as with "/" as argument whereas using "." as argument triggers the recursion.

comment:7 Changed 9 years ago by schwa

Hi Nicola,

I committed a changeset 876 which should fix the problem. The former "special case" is now the default as you had originally suggested; FTPHost._robust_ftp_command is simplified a bit.

Please get the corresponding version of ftputil.py, test it and give feedback. Note: You only need ftputil.py; the other changes are only to test code. Thanks for your help! :o)

Stefan

comment:8 Changed 9 years ago by schwa

Note: The patch should fix both problems, both ftp.gnome.org and ftp.sunfreeware.com should now behave as expected.

comment:9 Changed 9 years ago by ftputiluser

  • Resolution set to fixed
  • Status changed from assigned to closed

Thanks Stefan,

the fix seems fine,

Nicola

comment:10 Changed 9 years ago by schwa

  • Milestone set to 2.4.2
Note: See TracTickets for help on using tickets.