ftputil fails when server requires Windows-style slashes

[tephyr]

Client (my system)

• Linux grig 2.6.12-9-amd64-generic #1 Mon Oct 10 13:27:39 BST 2005 x86_64
• GNU/Linux
• Python 2.4.2

Server

• probably Windows
• 220-Wildcat! FTP Service (v6.0.451.1 Sat, 28 Jan 2006 13:18:33 -0800) ready.

The problem

The server admin decided to switch the "format" setting from unix to dos, and of course did not tell me, so my script failed. Their tech support said:

"There could be 1 thing - another customer had the same problem - it has to do with the format we are using - I was using unix style and change to dos - that is probably it. See if there is a setting you can try on your ftp program that changes the format."

So I used the host.set_directory_format('ms') command, but ftputil still failed with this error on the host (my script enters the path - it's not generated by ftputil):

listdir('\\Quotes_Daily\\daily_stocks') command:

PermanentError: 550 /\Quotes_Daily\daily_stocks: no such directory

It appears that the path.isabs method fails because it thinks '\' is not a valid absolute path, because the ftp_path module always delegates the path functions to posixpath.

I got it working with the following changes to ftp_path.py:

# Import ntpath
import posixpath, ntpath

    # force _Path.__init__ to use ntpath instead of posixpath
    def __init__(self, host):
        self._host = host
        # delegate these to the `posixpath` module
        pp = ntpath #posixpath

This obviously is a stopgap measure. The better solution would be for _Path.__init__ to check the host object for the directory format, and use posixpath or ntpath accordingly.

Notes

Stefan asked me to try this format: host.listdir('/Quotes_Daily/daily_stocks'), which did not work. Similar error to original.

[schwa]

I would like to have two pieces of information:

• a complete traceback when you call "host.listdir('/Quotes_Daily/daily_stocks')"
• a raw directory listing as it is returned by an FTP DIR command (you should be able to get this with an FTP command line client)

[tephyr]

Traceback:

Made connection to ?.com using ?/? combination
Will change to /Quotes_Daily/daily_stocks directory
Traceback (most recent call last):
  File "/home/aji/code/dev/python/misc/trunk/ftputil_repro5.py", line 17, in repro_1
    lstFTP = hstFTP.listdir(strdirchange)
  File "/usr/lib/python2.4/site-packages/ftputil/ftputil.py", line 634, in listdir
    return self._stat.listdir(path)
  File "/usr/lib/python2.4/site-packages/ftputil/ftp_stat.py", line 115, in listdir
    raise ftp_error.PermanentError("550 %s: no such directory" % path)
PermanentError: 550 /Quotes_Daily/daily_stocks: no such directory

Raw directory listing:

230-T.B.S.P. Inc. FTP Server v6.0.451.1 node: 1
230 User 71843itt logged in.
Remote system type is WIN32.
ftp> dir
200 PORT command successful.
150 Opening ASCII mode data connection for directory listing.
01-28-06  05:05AM       <DIR>          All_files
01-11-06  09:49PM       <DIR>          Monthly_Historical_Quotes
01-28-06  05:05AM       <DIR>          Quotes_Daily
01-11-06  04:47PM       <DIR>          6_Month_History_Files
01-11-06  04:44PM       <DIR>          Yearly_History_Files
01-28-06  12:24AM       <DIR>          Programs_For_All
226 Transfer complete.
ftp> dir /Quotes_Daily/daily_stocks
200 PORT command successful.
150 Opening ASCII mode data connection for directory listing.
01-28-06  04:48PM                 1024 00_index.txt
01-28-06  04:48PM                 1024 00_index.htm
01-03-03  05:43AM               246081 s010203.zip
01-03-04  04:55AM               295584 s010204.zip
01-04-03  05:43AM               245442 s010303.zip
01-04-05  05:06AM               204397 s010305.zip
01-04-06  04:59AM               322452 s010306.zip
...
01-01-05  05:07AM               246648 s123104.zip
226 Transfer complete.

There are ~800 files in that directory.

[anonymous]

The directory listing format is ok, i. e. should be recognized by the "ms" parser if it is set. If the listing format gets not detected automatically (probably not), set it manually with FTPHost.set_directory_format("ms"). Also, unless the FTP client translates the forward slashes to backward slashes before the DIR invocation, the server seems to handle forward slashes well.

The traceback is with posixpath used in ftp_path._Path's constructor, i. e. the version without your patch to use ntpath? From your traceback I see that self._path.isdir(path) runs without raising an exception, but the PermanentError is raised by the 'if' statement. Could you please add 'print' statements in _Stat.listdir, like in

    def listdir(self, path):
        """
        Return a list of directories, files etc. in the directory
        named `path`.
        """
        # we _can't_ put this check into `FTPHost._dir`; see its docstring
        print "*** Before abspath:", path
        path = self._path.abspath(path)
        print "*** After abspath:", path
        if not self._path.isdir(path):
            raise ftp_error.PermanentError("550 %s: no such directory" % path)
        ...

and give me the output?

[tephyr]

Correct, the traceback was with posixpath - my patch was not applied.

Here is the traceback with the print statements as requested.

Made connection to ?.com using ?/? combination
Will change to /Quotes_Daily/daily_stocks directory
*** Before abspath: /Quotes_Daily/daily_stocks
*** After abspath: /Quotes_Daily/daily_stocks
Traceback (most recent call last):
  File "/home/aji/code/dev/python/misc/trunk/ftputil_repro5.py", line 17, in repro_1
    lstFTP = hstFTP.listdir(strdirchange)
  File "/usr/lib/python2.4/site-packages/ftputil/ftputil.py", line 634, in listdir
    return self._stat.listdir(path)
  File "/usr/lib/python2.4/site-packages/ftputil/ftp_stat.py", line 117, in listdir
    raise ftp_error.PermanentError("550 %s: no such directory" % path)
PermanentError: 550 /Quotes_Daily/daily_stocks: no such directory

I added the FTPHost.set_directory_format("ms") as suggested - that gave me the proper output.

While trying to debug this, I added bad (extra and wrong direction) slashes to the path, so that even though the directory format was "ms", the path in my original code was invalid. Oops'''

This bug can be closed as is, although I suggest adding a different error message or having the code try to parse the directory list with an alternate parser. Telling me there's no such directory hid the actual problem in this case. Maybe "no such directory or cannot read the directory list"?

[schwa]

What about this error message?

"550 %s: no such directory or wrong directory parser used" % path

I hope that points the user to set_directory_format.

At first I had thought about "no such directory or wrong directory format". Though this describes the problem in words more related to the problem domain, it doesn't give a good hint about what the user of ftputil can do.

As I now think about it, set_directory_format possibly should have been set_directory_parser. Of course, I can't set the format the FTP server delivers; I can only set the parser in ftputil. And if the wrong parser is used, that still doesn't mean that something is inherently bad with the directory format used by the server.

Perhaps I will provide the alias set_directory_parser in the next minor release and drop set_directory_format in version 2.1.

[tephyr]

That's a much more helpful error message. It gets my vote.

[schwa]

Ok, checked in as changeset 464.