Opened 14 years ago

Closed 14 years ago

Last modified 14 years ago

#12 closed defect (fixed)

ftputil can't parse Unix directory format without user field

Reported by: Valeriy Pogrebitskiy (vpogrebi@… Owned by: schwa
Priority: major Milestone: 2.1
Component: Library Version: 2.1b2
Keywords: ftputil, DIR, LS, NLST, stat Cc:

Description

For some reason, ftplib.dir() command returns lines containing 9-member records (for most servers), and 8-member records - for other servers. When 8-member list is returned as a result of DIR command, ftp_stat.parse_line() fails (raises ParserError exception) when it tries to execute following statement:

try:

metadata, nlink, user, group, size, month, day, year_or_time, name = line.split(None, 8)

except ValueError:

# "unpack list of wrong size" raise ftp_error.ParserError?("line '%s' can't be parsed" % line )

Here is an example:

  1. Consider two <ftp_lib> connection objects connected to different servers
  2. ftp.host - is your FTPHost entity connected to SERVER1, and ftp1.host - FTPHost entity connected to SERVER2
  3. ftp.dir() (ftp1.dir()) command is executed for both objects. It fails for first, but succeeds for second connection:

ftp.dir('/export/home/vpogrebi')

Traceback (most recent call last):

File "<stdin>", line 1, in ? File "/usr/project/arena/prods/python/2.2.1/lib/python2.2/site-packages/ftp_lib/FTP.py", line 934, in dir

list = self.host.listdir(remote_dir)

File "/usr/project/arena/prods/python/2.2.1/lib/python2.2/site-packages/ftputil/ftputil.py", line 623, in listdir

file_list = self._stat.listdir(path)

File "/usr/project/arena/prods/python/2.2.1/lib/python2.2/site-packages/ftputil/ftp_stat.py", line 115, in listdir

raise ftp_error.PermanentError("550 %s: no such directory" % path)

ftputil.ftp_error.PermanentError: 550 /export/home/vpogrebi: no such directory

ftp1.dir('hedgefunds/dbmasters')

['inbound', 'outbound']

Attempt to use self.listdir() fails for first connection, but succeeds for second as well:

ftp.host._stat.listdir(remote_dir)

Traceback (most recent call last):

File "<stdin>", line 1, in ? File "/usr/project/arena/prods/python/2.2.1/lib/python2.2/site-packages/ftputil/ftp_stat.py", line 115, in listdir

raise ftp_error.PermanentError?("550 %s: no such directory" % path)

ftputil.ftp_error.PermanentError: 550 /export/home/vpogrebi/arena_rcs: no such directory

ftp1.host.listdir('hedgefunds/dbmasters')

['inbound', 'outbound']

Research showed that actual exception is being raised by ftp_stat.parse_line() function call when following statement fails:

metadata, nlink, user, group, size, month, day, year_or_time, name = line.split(None, 8)

Further research reveals that <line> passed to ftp_stat.parse_line() is being SPLIT onto 8-member list in case of SERVER1, and 9-member list in case of SERVER2. This last fact is in turn a direct result of the fact that <ftplib.dir()> returns different results for these two servers:

lines = ftp.host._stat._host_dir(dirname) lines

['total 49712', '-rwxrwxrwx 1 staff 23469 Jul 12 15:07 %backup%~', '-rw-r--r-- 1 staff 3015 May 5 16:08 .bash_aliases', '-rw-r--r-- 1 staff 3015 Apr 27 15:51 .bash_aliases~', '-rw-r----- 1 staff 12635 Jul 11 16:51 .bash_history', '-rw-r--r-- 1 staff 373 Mar 17 2003 .bashrc', '-rw-r--r-- 1 staff 287 Mar 6 2003 .bashrc~', '-rw-r----- 1 staff 61486 Apr 19 11:25 .emacs', '-rw-r----- 1 staff 2817 Nov 9 2001 .emacs_tmp', '-rw-r----- 1 staff 61358 Sep 16 2002 .emacs~', 'drwx------ 2 staff 96 Apr 29 09:56 .gnupg', '-rwxr--r-- 1 staff 2386 Mar 5 2003 .ksh_aliases', '-rwxr--r-- 1 staff 2384 Mar 5 2003 .ksh_aliases~', '-rwxr--r-- 1 staff 431 Sep 16 2002 .kshrc', '-rwxr--r-- 1 staff 377 Sep 12 2002 .kshrc~', '-rw-r--r-- 1 staff 676 Nov 8 2001 .login', '-rw-r----- 1 arena 105 Mar 24 16:15 .mailcap', '-rw-r----- 1 arena 230 Mar 24 14:15 .mime.types', '-rw-r--r-- 1 staff 1167 Dec 8 2003 .profile', '-rw-r--r-- 1 staff 536 Nov 8 2001 .profile_arena', '-rw-r--r-- 1 staff 1223 Dec 8 2003 .profile~', '-rw------- 1 arena 1024 Apr 15 11:24 .rnd', '-rw------- 1 staff 3936 Jul 11 13:08 .sh_history', 'drwxrwxrwx 2 staff 1024 May 13 14:30 .ssh', '-rwxr--r-- 1 staff 611 Mar 26 2003 .startxst', '-rwxr--r-- 1 staff 520 Mar 6 2003 .startxst~', '-rw------- 1 staff 893 Jul 11 13:06 .TTauthority', '-rw------- 1 staff 208 Jun 20 2002 .Xauthority', '-rw-r----- 1 staff 2296 Dec 10 2002 .xemacs-options', '-rw-r--r-- 1 staff 2296 Nov 8 2001 .xemacs-options-current', '-rw-r----- 1 staff 60773 Nov 9 2001 .xemacs-options-new', '-rw-r----- 1 staff 60671 Nov 9 2001 .xemacs-options-new~', '-rw-r--r-- 1 staff 2296 Nov 8 2001 .xemacs-options-orig', '-rw-r----- 1 staff 2315 Sep 17 2002 .xemacs-options.20020917', '-rw-r----- 1 staff 60774 Nov 12 2001 .xemacs-options~', 'drwxr-x--- 18 staff 1024 Jul 14 10:04 arena_rcs', 'drwxrwxrwx 2 staff 1024 Apr 16 16:55 bin', '-rw-rw-r-- 1 staff 301 Mar 25 2003 DB_TEST.txt', '-rw-rw-rw- 1 staff 296 May 27 15:57 DBARS.EquityData?.eqf', 'drwxrwxrwx 25 staff 2048 Apr 27 13:42 DEAM', '-rw-r----- 1 staff 0 Jul 7 09:39 fapr-fa-bx-DWS_BOX.jil', '-rwxr--r-- 1 staff 716 Feb 12 2002 FONT_NAMES', '-rw-r--r-- 1 staff 24939560 May 10 09:16 gdh_data_load.RATINGS.miss.20040509', '-rw-r----- 1 staff 1782 Mar 16 13:33 Hedge.DWS_TRADES.20040105.4424703.xml.test', 'drwxr-xr-x 13 staff 1024 Jun 14 10:16 install', 'drwxr-xr-x 7 staff 1024 Dec 3 2003 Python', '-rw-r----- 1 staff 17616 Mar 16 14:56 Ready*', '-rwxr--r-- 1 staff 17375 Nov 9 2001 rgb.txt.orig', '-rw-r----- 1 staff 0 May 26 11:33 RIC_Err.txt', '-rwxrwxrwx 1 staff 44241 May 14 11:09 RMClient', '-rw-rw-r-- 1 staff 192 Mar 24 10:02 RMClient_InstallLog.log', 'drwxr-xr-x 3 staff 1024 May 6 16:14 setup', '-rwxrwxrwx 1 staff 21 May 14 11:04 SFTP_TEST', '-rw-rw-rw- 1 arena 1928 May 13 11:43 sftp_test.log', '-rw-rw-rw- 1 staff 3010 Jun 2 2003 SIGNAL.man', '-r-xr-xr-x 1 arena 2169 Jan 28 2003 site-startup.ael', 'drwxr-xr-x 2 staff 96 Oct 10 2002 trd_aggr', 'drwxr-xr-x 3 staff 96 Sep 24 2002 usr', '-rwxr--r-- 1 staff 4843 Jan 15 2002 WAITPID.man']

lines = ftp1.host._stat._host_dir('hedgefunds/dbmasters') lines

['total 18', 'drwxr-x--- 5 1105 150 9216 Jul 14 17:54 inbound', 'drwxr-x--- 4 1105 150 9216 Jul 14 15:05 outbound']

From comparing output of DIR command and 'ls -l' executed on the Unix box, it appears that DIR output 'loses' USER ID... :

So, in the end the fact that DIR returns 8-member list results in attempt to parse that list into 9 values within ftp_stat.parse_line() fails:

line.split(None, 8)

['drwxr-x---', '18', 'staff', '1024', 'Jul', '14', '10:04', 'arena_rcs']

line.split(None, 9)

['drwxr-x---', '18', 'staff', '1024', 'Jul', '14', '10:04', 'arena_rcs']

metadata, nlink, user, group, size, month, day, year_or_time, name = line.split(None, 9)

Traceback (most recent call last):

File "<stdin>", line 1, in ?

ValueError?: unpack list of wrong size

metadata, nlink, user, group, size, month, day, year_or_time, name = line.split(None, 8)

Traceback (most recent call last):

File "<stdin>", line 1, in ?

ValueError?: unpack list of wrong size

It appears that use of NLST instead of DIR solves the problem.

NOTICE: This behavior observed in one of the earlier releases of FTPUTIL. I am not sure if it still persists.

Change History (3)

comment:1 Changed 14 years ago by Valeriy Pogrebitskiy (vpogrebi@…

Keywords: FTPUTIL DIR LS NLST STAT added

comment:2 Changed 14 years ago by schwa

Keywords: ftputil stat added; FTPUTIL STAT removed
Resolution: fixed
Status: newclosed
Summary: DIR vs. NLSTftputil can't parse Unix directory format without user field
Version: 2.1b2

In commit 517, I modified the Unix parser to deal with the described behavior. It will fail in the rare case that a real file name starts with "-> " which should be quite unlikely. As the ticket submitter told me that he hasn't access to the unusual server anymore, my fix couldn't be tested in the field though I added some automated tests to check the new code.

I haven't used the NLST command because I'm not so sure that it's always available and yields the same result/format for server with Unix-style directory format. To reflect this I've changed the summary.

comment:3 Changed 14 years ago by schwa

Milestone: 2.1
Note: See TracTickets for help on using tickets.